PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Distilled RLVR

Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, Nan Duan

arXiv:2604.03128v2cs.LGcs.CL

TL;DR

OPSD’s privileged teacher can produce information leakage and unstable long-term training, motivating a method that separates useful token-level modulation from update direction. RLSD uses environmental rewards for direction and self-distillation for magnitude, achieving the highest average accuracy across five multimodal reasoning benchmarks while outperforming the Base LLM by 4.69%.

  • Problem

    OPSD’s information-asymmetric teacher-student setup can cause privileged information leakage and unstable long-term training.

  • Method

    RLSD uses reliable environmental rewards to determine update directions and the privileged teacher’s evidence ratio to modulate dense token-level update magnitudes.

  • Results

    RLSD achieves the highest average accuracy across five multimodal reasoning benchmarks, outperforming the Base LLM by 4.69% and GRPO by 2.32%.

  • Takeaways & Limitations

    Self-distillation can supply fine-grained credit information while RLVR anchors optimization to environmental feedback.

  • Takeaways & Limitations

    The reported experiments are limited primarily to multimodal reasoning scenarios, with broader settings deferred to a forthcoming version.

Abstract

from arXiv · show

On-policy distillation (OPD) has become a popular training paradigm in the LLM community. This paradigm selects a larger model as the teacher to provide dense, fine-grained signals for each sampled trajectory, in contrast to reinforcement learning with verifiable rewards (RLVR), which only obtains sparse signals from verifiable outcomes in the environment. Recently, the community has explored on-policy self-distillation (OPSD), where the same model serves as both teacher and student, with the teacher receiving additional privileged information such as reference answers to enable self-evolution. This paper demonstrates that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training. Accordingly, we identify the optimal niche for self-distillation and propose \textbf{RLSD} (\textbf{RL}VR with \textbf{S}elf-\textbf{D}istillation). Specifically, we leverage self-distillation to obtain token-level policy differences for determining fine-grained update magnitudes, while continuing to use RLVR to derive reliable update directions from environmental feedback (e.g., response correctness). This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.

1 Introduction

The introduction identifies information asymmetry as the source of OPSD’s leakage and degradation, then proposes RLSD to retain dense self-distillation signals while anchoring update directions in environmental rewards.

  • OPD uses dense token-level teacher signals, whereas RLVR supplies only a scalar outcome signal for each trajectory.
  • OPSD avoids an external teacher but its privileged conditioning causes early performance peaks, later deterioration, and leakage of inaccessible reference information.
  • Information asymmetry makes OPSD distribution matching ill-posed through an irreducible mutual-information gap and r-specific gradient deviations.
  • The teacher’s evidence ratio is retained as a dense token-level magnitude signal because it measures how privileged information revises token beliefs.
  • RLSD uses environmental rewards to determine update direction and the privileged teacher’s evidence ratio to modulate update magnitude.
  • Across five multimodal reasoning benchmarks, RLSD achieves the highest average accuracy, exceeding the Base LLM by 4.69% and GRPO by 2.32%.

2 Preliminaries

This section introduces GRPO, OPD, and OPSD, emphasizing their different supervision sources and the limitation of sequence-level advantages that motivates token-level credit assignment.

  • GRPO: GRPO samples response groups and uses verifier rewards to compute sequence-level advantages for policy updates.
  • GRPO: Because rewards are provided only at the sequence level, GRPO assigns the same advantage to every token in a response.
  • OPD and OPSD: OPD has a separate larger teacher provide dense token-level supervision on trajectories sampled by the student, but requires substantial computational overhead.
  • OPD and OPSD: OPSD uses one model as both teacher and student, with the teacher conditioned on privileged information unavailable to the student.
  • Figure 2 illustrates OPSD leakage through a model’s appeal to an invisible reference solution during inference.
  • Figure 3 compares leakage, KL divergence, and validation performance across OPSD and ablated variants.
  • OPD and OPSD: OPSD minimizes per-token divergence between teacher and student distributions while backpropagating gradients only through the student.

3 Why Does OPD Work While OPSD Fails?

OPSD fails because its privileged teacher target is information-asymmetric with the student, creating an irreducible gap that contaminates per-sample gradients and drives leakage and degradation.

  • Empirical diagnosis: Empirically, OPSD leakage rises over training, validation accuracy peaks within 10–20 steps before declining, and teacher–student KL divergence stagnates.The KL plateau is consistent with the irreducible objective gap, while later degradation tracks increasing reliance on inaccessible information.
  • Structural mismatch: OPSD matches a student distribution that cannot condition on r to a teacher distribution that does, making the objective fundamentally ill-posed.The ideal student recovers the marginal teacher distribution over privileged information, whereas OPSD enforces per-sample matching.
  • Irreducible gap: The irreducible conditional mutual-information gap I(Yt; R | X, Y<t) > 0 is independent of student parameters and cannot be eliminated by optimization.Its residual loss is a strictly positive lower bound that grows with the informativeness of the privileged signal.
  • Gradient structure: Although the expected OPSD gradient matches ideal marginal matching, concrete per-sample gradients contain a zero-mean deviation whose variance increases with privileged-information mutual information.This explains why population-level analysis can appear benign while sampled optimization remains pathological.
  • Training dynamics: Early training improves validation accuracy, but as the student approaches the teacher marginal, deviation terms dominate and updates encode x → r correlations.This two-phase dynamic corresponds to initial accuracy gains followed by performance decline and increasing leakage.
  • Leakage bandwidth: Leakage increases across OPSD variants, while concentrated teacher targets produce more severe leakage despite narrower effective bandwidth.Full OPSD has the widest bandwidth; Teacher’s Top-1 concentrates privileged information most strongly, and Student’s Top-1 has the lowest leakage rate.

4 RLSD: Self-Distillation as RLVR’s Wingman

RLSD repurposes privileged self-distillation as token-level credit modulation while keeping verifier rewards responsible for update direction. It therefore preserves dense discrimination without distribution matching or auxiliary distillation loss.

  • Core design: RLSD uses the verifier-derived advantage to determine whether tokens are reinforced or penalized, while the teacher signal determines their relative update magnitudes.This decouples update direction from privileged information while retaining fine-grained token-level credit assignment.
  • Token signal: The privileged information gain Δt compares teacher and student token log-probabilities, measuring how strongly r supports or disfavors each sampled token.Positive Δt indicates support from privileged information; negative Δt indicates disfavour.
  • Direction-aware weighting: Direction-aware reweighting uses PT /PS for positive advantages and PS /PT for negative advantages, increasing credit for supported correct tokens and blame for disfavored incorrect tokens.The exponentiation produces strictly positive weights, preserving the reward-defined update sign.
  • Stabilization: RLSD clips evidence weights to bound each token’s credit influence, paralleling GRPO’s trust-region control of policy-update step size.The method also interpolates from uniform to reweighted advantages using λ ∈[0, 1] to avoid an abrupt training transition.
  • Pipeline: RLSD requires no auxiliary distillation loss; its only standard-GRPO modification is internal credit redistribution, with one additional teacher-logit forward pass per response.The added pass is described as negligible relative to rollout generation.
  • Positioning: Unlike GRPO’s uniform token advantage and OPSD’s teacher-only distribution matching, RLSD combines verifier-grounded direction with teacher-based relative credit.This provides targeted token-level assignment without replacing correctness feedback.

5 Experiment

Experiments evaluate RLSD on five multimodal reasoning benchmarks using Qwen3-VL-8B-Instruct. RLSD achieves the strongest average accuracy, improves challenging mathematical reasoning, and shows faster, more stable training dynamics than key baselines.

  • Experimental setup: Training uses the challenging MMFineReason-123K subset and evaluates multimodal mathematical and general reasoning across five benchmarks.The training subset retains samples where Qwen3-VL-4B-Thinking fails across four independent rollouts.
  • Main results: RLSD achieves the highest average accuracy across five multimodal reasoning benchmarks, outperforming the Base LLM by 4.69% and GRPO by 2.32% under 4K.It also consistently outperforms OPSD and SDPO.
  • Main results: RLSD improves MathVista by +1.9% and MathVision by +3.91% over GRPO, where fine-grained reasoning-step discrimination matters most.It also exceeds the additive GRPO+OPSD baseline by 3.27 points.
  • Training dynamics: RLSD shows a steeper initial ascent and a higher accuracy reward ceiling than GRPO while avoiding OPSD’s late-stage performance collapse.The dynamics are shown over 200 optimization steps.
  • Training dynamics: RLSD maintains higher entropy than GRPO by selectively strengthening critical reasoning tokens instead of uniformly suppressing alternatives.GRPO’s entropy collapse is attributed to its uniform sequence-level reward.
  • Credit assignment: RLSD assigns larger credit to decisive tokens in correct trajectories and strongest blame to misread relations and derived wrong answers in incorrect trajectories.Generic narration and neutral setup tokens receive comparatively less credit or blame.

6 Related Work

Prior work uses verifier rewards, teacher distillation, or heuristic token-credit proxies to improve reasoning. RLSD differs by rejecting privileged distribution matching and using privileged information only for credit magnitude.

  • Verifier-based RL: GRPO uses verifier-only sequence-level rewards, giving every token identical advantage regardless of its contribution to correctness.This motivates finer-grained credit assignment for long-horizon reasoning.
  • Token-level credit: Verifier-only credit-assignment methods estimate token importance from intrinsic proxies such as entropy, uncertainty, attention, or outcome sensitivity.These approaches remain distinct from RLSD’s privileged-information-based mechanism.
  • On-policy distillation: OPD supplies dense token-level supervision from a separate larger teacher but requires maintaining that teacher throughout training.The separate teacher introduces computational overhead.
  • RLSD distinction: RLSD departs from privileged teacher–student distribution matching because inaccessible latent information makes that objective structurally ill-posed and can induce leakage and unstable long-run training.Its design retains the teacher signal without using it as a generative target.

7 Conclusion

The paper concludes that OPSD is structurally limited by information asymmetry, while RLSD uses self-distillation for token-level update magnitudes and environment rewards for update directions.

  • 7 Conclusion: OPSD’s information asymmetry makes distribution matching ill-posed, causing privileged-information leakage and progressive performance degradation.The analysis links this failure to an irreducible mutual information gap and the observed pattern of early gains followed by collapse.
  • 7 Conclusion: RLSD repurposes the discrepancy between prior and posterior policies to control per-token update magnitudes rather than forcing the student to imitate the teacher distribution.This preserves fine-grained credit control while avoiding direct distribution matching with the privileged teacher.
  • 7 Conclusion: RLSD anchors update directions to environmental rewards while retaining OPSD’s token-level supervision.The framework therefore combines self-distillation’s dense supervision with RLVR’s environmental grounding.

8 Limitations and Future Work

The paper primarily develops and validates RLSD through theoretical analysis and limited experiments focused on multimodal reasoning, while broader evaluations are deferred.

  • 8 Limitations and Future Work: The paper focuses primarily on theoretical analysis of OPSD’s structural limitations and RLSD’s motivation and validation.This establishes the paper’s main scope rather than presenting a broad empirical study.
  • 8 Limitations and Future Work: The current version provides limited experiments centered on multimodal reasoning scenarios.The authors report preliminary validation in pure text reasoning, video understanding, and additional model families, with those results reserved for a forthcoming version.

A.1 Proof of Theorem 1 (KL Decomposition)

The KL objective decomposes into an information-theoretic term measuring privileged information and a remaining student-dependent term.

  • A.1 Proof of Theorem 1 (KL Decomposition): The expected KL divergence E_r[D_KL(P_T(· | r) ∥ P̄_T)] equals the conditional mutual information I(Y_t; R | X, Y_<t).This identifies the irreducible information gap contributed by conditioning the teacher on privileged information.
  • A.1 Proof of Theorem 1 (KL Decomposition): The OPSD loss decomposes as L_OPSD = L* + I(Y_t; R | X, Y_<t).The first term is identified with the expected KL divergence between the privileged teacher distribution and its marginal distribution.

A.2 Proof of Proposition 1 (Per-Sample Gradient Decomposition)

The per-sample OPSD gradient separates into an expected gradient and a privileged-information-dependent residual whose magnitude is governed by teacher-distribution variance.

  • A.2 Proof of Proposition 1 (Per-Sample Gradient Decomposition): The per-sample gradient decomposes as g(θ; r) = g*(θ) + δ(θ; r).The decomposition is obtained by adding and subtracting the marginal teacher distribution P̄_T(v) inside the sum.
  • A.2 Proof of Proposition 1 (Per-Sample Gradient Decomposition): The expected teacher distribution equals its marginal, and the student score gradient is independent of the privileged variable r.These properties establish the expected-gradient component while isolating the residual dependence on privileged information.
  • A.2 Proof of Proposition 1 (Per-Sample Gradient Decomposition): The residual’s diagonal contribution is proportional to Var_r[P_T(v | r)] · ∥∇_θ log P_S(v)∥², with cross terms bounded by the same variances.The stated result is exact for the diagonal contribution and a lower bound for the full expression.
  • A.2 Proof of Proposition 1 (Per-Sample Gradient Decomposition): When I(Y_t; R | X) = 0, the privileged teacher distribution equals its marginal for every r, so the residual δ vanishes.In that case, privileged information does not induce the additional gradient component.

A.3 Proof of Irreducibility of Leakage Across Pilot Variants

Across all tested distribution-matching variants, privileged information remains embedded in update directions, making leakage irreducible. Teacher management also creates an instability trade-off, whereas RLSD isolates privileged information in a bounded scalar weight while preserving environment-determined update directions.

  • Leakage irreducibility: All three pilot variants retain privileged-information dependence in the gradient direction, regardless of how the distillation target is compressed.Their deviation component has strictly positive variance whenever I(Y_t; R | X) > 0.
  • Training instability: Online teacher updates can worsen the true objective even when each step descends its local surrogate.This occurs when teacher drift exceeds student progress, with positive teacher drift.
  • Teacher management: The impossibility trilemma shows that shared-parameter distribution matching cannot simultaneously ensure objective stability, sustained improvement, and leakage-free training.Hybrid snapshot strategies only interpolate between stability and signal persistence; the mutual-information gap remains.
  • Teacher management: Frozen teachers prevent objective drift but eventually make the distillation signal vanish, limiting the student to the initial checkpoint’s quality.The frozen-teacher strategy satisfies stability while violating sustained improvement.
  • RLSD resolution: The evidence ratio provides token-level credit attribution by measuring how each student-generated token updates belief in the privileged information.Weights decompose sequence-level evidence into token contributions without requiring imitation of the teacher’s exact wording.
  • RLSD resolution: RLSD uses the environment reward to determine update direction and the privileged evidence ratio only to modulate bounded update magnitude.Its design isolates direction and support from privileged information, clips the weight, and converges toward vanilla GRPO as PS approaches PT.
Loading 2604.03128v2…