PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilled Reinforcement Learning for LLM Post-training

Chen Wang, Zhaochun Li, Jionghao Bai, Yining Zhang, Hexuan Deng, Ge Lan, Yue Wang

arXiv:2607.17247v1cs.LGcs.AI

TL;DR

Existing reinforcement learning and on-policy distillation methods provide either coarse outcome supervision or unconditional teacher imitation, limiting credit assignment and knowledge transfer. Distilled RL integrates selective teacher guidance into the reinforcement learning objective and outperforms standard RL and OPD across within-family and cross-family distillation on Pass@1 and Pass@k.

  • Problem

    Reinforcement learning uses coarse sequence-level rewards, while on-policy distillation unconditionally imitates teacher distributions, limiting credit assignment and knowledge acquisition.

  • Method

    Distilled RL integrates teacher preferences into the RL objective through token-level reverse importance weighting, negative-sample reset, and sequence-level normalization.

  • Results

    Across within-family and cross-family distillation, Distilled RL consistently outperformed RL and OPD on Pass@1 and Pass@k; DSQW-1.5B average Pass@1 rose from 31.70 to 40.00.

  • Takeaways & Limitations

    Distilled RL enables selective fine-grained transfer of previously unavailable teacher knowledge while preserving reinforcement-learning optimization behavior.

  • Takeaways & Limitations

    The method assumes that when students solve problems correctly, teachers are sufficiently competent for their token-level preferences to provide useful guidance.

Abstract

from arXiv · show

Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.

1 INTRODUCTION

The introduction motivates Distilled RL by identifying coarse-grained RL supervision and limited knowledge acquisition as key limitations. It then presents a framework that injects teacher knowledge into the RL gradient through three components and reports evidence from a controlled case study and broader distillation experiments.

  • LLM post-training improves reasoning, instruction following, and task adaptation, while RL updates trajectories sampled from the current policy to reduce training–inference distribution shift.
  • Sequence-level rewards create severe credit-assignment problems because only a small subset of tokens may determine answer correctness.This limits information about which intermediate reasoning steps should be encouraged or corrected.
  • RL mainly reinforces behaviors already accessible to the student and has limited ability to introduce knowledge absent from its current capabilities.
  • Distilled RL incorporates teacher knowledge directly into the RL gradient by using reverse importance ratios to redistribute learning signals at the token level.Its three components are reverse importance sampling, negative sample reset, and sequence-level geometric normalization.
  • A controlled case study demonstrates that Distilled RL enables students to acquire new teacher knowledge that standard RL alone cannot learn.
  • Extensive experiments evaluate Distilled RL in both within-family and cross-family distillation settings.

2 RELATED WORK

Related work centers on reinforcement learning for improving LLM reasoning and on-policy distillation for transferring teacher capabilities along student-generated trajectories. Recent methods include verifiable-reward RL, GRPO, reverse-KL distillation, token-level supervision, and hybrid or uncertainty-aware objectives.

  • Reinforcement Learning for LLMs: RLHF improves LLM reasoning by optimizing human-preference reward models with policy-gradient algorithms such as PPO.Recent reinforcement learning with verifiable rewards instead directly evaluates responses using rule-based outcome signals.
  • Reinforcement Learning for LLMs: GRPO estimates advantages from relative rewards among grouped responses without a separate critic, while large-scale RL can elicit sophisticated reasoning behaviors.On-policy RL also reduces training–inference distribution mismatch by training on trajectories generated by the evolving policy.
  • On-Policy Distillation: On-policy distillation samples responses from the student and transfers teacher capabilities by matching teacher-generated outputs or predictive distributions.Conventional sequence-level distillation uses fixed teacher-generated data and can suffer exposure bias on uncovered student states.
  • On-Policy Distillation: MiniLLM uses reverse KL with student-generated samples, while generalized knowledge distillation formalizes token-level teacher supervision along student-generated trajectories.Subsequent work explores alternative KL objectives, uncertainty-aware distillation, and combinations of knowledge distillation with reinforcement learning.

3 METHOD

Distilled RL couples teacher supervision with reward-driven RL rather than unconditional KL imitation, using selective token-level importance weighting to transfer useful knowledge. Its design combines reverse importance sampling, negative-sample reset, and sequence-level geometric normalization to avoid misleading or globally suppressive teacher guidance.

  • Motivation: OPD improves quickly but soon saturates or declines, especially in cross-family distillation, whereas RL improves more slowly but sustains an upward trend.OPD aligns the student with the teacher on visited states but provides limited task-aware signals for continued improvement.
  • Motivation: Larger OPD rollout groups G ∈{1, 4, 8} produce similar performance, indicating that KL-based imitation—not sampling variance—is the main bottleneck.Additional on-policy samples cannot overcome the local optimum induced by teacher–student distribution matching.
  • Distilled RL objective: Distilled RL integrates teacher guidance directly into the RL objective, using teacher likelihoods on student-generated tokens to redistribute the learning signal through token-level importance weights.This replaces an independent KL-based imitation loss with selective, normalized importance weighting.
  • Negative sample reset: For positive-advantage samples, teacher-preferred tokens with ρi,t > 1 receive stronger rewards, while negative-advantage samples reset the importance weight to one and retain the original RL update.The reset prevents teacher guidance from pushing the student away from a teacher preference that should be imitated.
  • Sequence-level geometric normalization: Sequence-level geometric normalization offsets multiplicative token-ratio effects so teacher guidance redistributes learning across tokens instead of globally amplifying or suppressing a response.This addresses frequently sub-unit student-to-teacher ratios that would otherwise over-suppress successful trajectories.

4 INTERPRETABLE TEACHER-SIDE INFORMATION TRANSFER

This section uses an interpretable entropy case study to test whether Distilled RL can transfer teacher-side distributional information beyond task rewards. A temperature-controlled teacher shows that the student captures and maintains the teacher’s entropy characteristic.

  • Case study design: Distilled RL is evaluated through entropy because capturing properties of the teacher’s distribution is necessary for learning new teacher knowledge.The case study tests teacher-side information transfer beyond task rewards without relying on explicit distribution matching.
  • Case study design: A temperature-controlled teacher uses temperature 0.8 when student entropy exceeds 0.5 and temperature 1.2 otherwise, isolating distributional transfer from model scale and architecture.The teacher is constructed from the student model itself, providing sharper or softer signals based on the student’s entropy.
  • Entropy transfer result: The student entropy rapidly approaches the target region and fluctuates around the threshold, demonstrating that Distilled RL captures and transfers the teacher distribution’s entropy characteristic.The low-temperature teacher suppresses excessive uncertainty when entropy is high, while the high-temperature teacher encourages a softer distribution when entropy is low.

5 EXPERIMENTS

Experiments across within-family and cross-family settings show that Distilled RL consistently outperforms OPD, RL, and OPD+RL on Pass@1 and Pass@k, while maintaining stable training dynamics and generalization. Ablations identify negative sample reset and sequence-level geometric normalization as important components, with negative sample reset having the largest impact.

  • Experimental Setup: Experiments use DSQW-1.5B, Qwen3-4B, and Qwen3-1.7B with DAPO-17K, evaluating mathematical reasoning and knowledge-intensive generalization benchmarks.Distilled RL is implemented in the EasyR1 and VeRL frameworks and compared against OPD, RL (GRPO), and OPD+RL.
  • Performance: 40.00 average Pass@1 is achieved by Distilled RL on DSQW-1.5B, improving from 31.70 and surpassing OPD, RL, and OPD+RL by 4.73, 3.14, and 3.46 points.On Qwen3-1.7B, Distilled RL reaches 46.37 average Pass@1; on Qwen3-4B, it improves from 46.33 to 58.96.
  • Training Dynamics: Distilled RL maintains higher rewards, moderate and stable policy entropy, stable response length, and steadily improving average Pass@1 during training.These dynamics indicate more stable optimization than the compared baseline methods across the evaluated model architectures.
  • Generalization: Distilled RL achieves the best or highly competitive performance on MMLU-Pro and SuperGPQA and also performs strongly under Pass@16 evaluation.These results indicate that the objective does not overfit to mathematical reasoning and preserves general knowledge capabilities during post-training.
  • Cross-family Distillation: Cross-family distillation particularly favors Distilled RL because OPD remains behind RL and OPD+RL does not consistently outperform RL on DSQW-1.5B.The passage attributes this limitation to unconditional KL-based teacher imitation interfering with reward optimization when teacher and student reasoning distributions differ substantially.
  • Ablation: 8.81 and 6.39 points are the average Pass@1 decreases from removing negative sample reset on Qwen3-4B and DSQW-1.5B, respectively.Removing sequence-level geometric normalization also degrades performance, because token-level importance ratios below one can shrink rapidly across student-generated sequences.

6 CONCLUSION

Distilled RL integrates teacher supervision directly into the reinforcement learning objective, using three mechanisms to enable fine-grained knowledge transfer while preserving reinforcement-learning optimization behavior. An entropy-control case study and experiments across within-family and cross-family settings demonstrate transfer of previously unavailable teacher knowledge beyond the original reward signal.

  • Framework: Distilled RL integrates teacher supervision directly into the reinforcement learning objective, unlike conventional on-policy distillation’s unconditional KL-based imitation.It uses reverse importance sampling to selectively redistribute the RL learning signal according to teacher preference.
  • Framework: The framework combines reverse importance sampling, negative sample reset, and sequence-level geometric normalization.Together, these mechanisms enable fine-grained knowledge transfer while preserving reinforcement-learning optimization behavior.
  • Results: An interpretable entropy-control case study showed that Distilled RL transfers previously unavailable teacher knowledge to the student beyond the original reward signal.The conclusion reports this as evidence from an entropy-control case study.

Appendix · A LIMITATION

On-policy distillation cannot directly establish whether the teacher can solve a problem because it evaluates only student-generated trajectories. Distilled RL mitigates this risk through negative sample reset, but an implicit assumption remains.

  • A LIMITATION: On-policy distillation queries the teacher only on student-generated trajectories, without sampling complete teacher responses.This prevents direct assessment of the teacher’s ability to solve a given problem.
  • A LIMITATION: Negative sample reset mitigates this limitation by removing teacher-based reweighting from negative student trajectories.
  • A LIMITATION: Despite this mitigation, the training procedure retains an implicit assumption.The supplied passage does not specify the assumption’s full content.

B IMPLEMENTATION DETAILS

The experiments use standardized Distilled RL training with Qwen3-8B-GRPO supervising Qwen3-4B and DSQW-1.5B students. Evaluation spans mathematical reasoning benchmarks and knowledge-intensive generalization tests excluded from supervised training.

  • Experimental setup: Distilled RL is implemented with EasyR1 and VeRL on DAPO-17K under shared prompt distributions, rollout budgets, optimization schedules, and evaluation protocols.Qwen3-8B-GRPO serves as teacher for Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B (DSQW-1.5B).
  • Experimental setup: Each prompt produces G = 8 student responses, receives binary correctness rewards, and uses within-group reward normalization to compute response-level advantages.The procedure follows GRPO.
  • Evaluation benchmarks: Mathematical reasoning is evaluated on ten benchmarks, including AIME24, AIME25, AIME26, CMIMC25, HMMT25, AMC23, GSM8K, MATH500, Minerva Math, and Olympiad.The first five also provide the averaged competition score used in training-dynamics figures.
  • Evaluation benchmarks: Post-training generalization is assessed on MMLU-Pro and a fixed 500-example SuperGPQA subset, with neither benchmark contributing supervised training targets.The subset addresses SuperGPQA’s relatively large evaluation cost.
Loading 2607.17247v1…