HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation

Ken Ding

Source-linked AI summary

HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation

Ken Ding

arXiv:2603.23871v1Mar 25, 2026cs.LGcs.AI

TL;DR

RL training can receive no gradient on mathematical prompts where every rollout fails. HDPO addresses these cliff prompts with privileged self-distillation from the same model, and experiments show improved coverage with λ controlling the tradeoff against greedy accuracy.

Problem
RL policy gradients vanish on cliff prompts where all rollouts fail, leaving these difficult mathematical problems without a learning signal.
Method
HDPO identifies cliff prompts, generates ground-truth-conditioned rollouts, filters correct solutions, and distills the same model’s teacher distribution into the student.
Results
HDPO consistently improves pass@4 and pass@8, with λ controlling the exploration–exploitation tradeoff; at λ=0.1, pass@8 improves by +1.4–1.7%.
Takeaways & Limitations
Same-model privileged distillation provides bounded realizability and a way to supply learning signal on prompts where standard RL gradients vanish.
Takeaways & Limitations
Empirical results are demonstrated only at 1.5B parameters on OpenMathInstruct-2, and HDPO adds overhead proportional to the number of cliff prompts.

Abstract

from arXiv · show

Large language models trained with reinforcement learning (RL) for mathematical reasoning face a fundamental challenge: on problems the model cannot solve at all - "cliff" prompts - the RL gradient vanishes entirely, preventing any learning signal from reaching these failure modes. We introduce Hybrid Distillation Policy Optimization (HDPO), which augments standard RL with privileged self-distillation targeting cliff prompts. On each training step, HDPO identifies prompts where all rollouts fail, generates privileged rollouts by providing the model with ground-truth information, filters for correct solutions, and distills the teacher's token-level distribution into the student. Because teacher and student share the same weights - differing only in their input - the realizability gap is provably bounded, unlike cross-model distillation. We prove that R=1 filtered privileged generation recovers the optimal KL-regularized RL policy in the hard-threshold limit. Experiments on OpenMathInstruct-2 with Qwen2.5-Math-1.5B-Instruct show that HDPO consistently improves coverage metrics (pass@4 by +0.8-1.1%, pass@8 by +0.4-1.7%) while maintaining greedy accuracy, with the distillation weight lambda providing direct control over the exploration-exploitation tradeoff.

1 Introduction

HDPO addresses the zero-gradient failure of RL on cliff prompts by combining standard RL with privileged self-distillation. It uses the same model as teacher and student, with theory supporting its realizability and filtered-generation targets, and experiments showing broader coverage.

Motivation: GRPO provides no learning signal when all rollouts for a prompt fail, creating a dead zone precisely where learning is most needed.Identical zero rewards produce identical advantages, so the policy gradient vanishes on these cliff prompts.
HDPO: HDPO uses the same model as teacher and student, differing only in input, which bounds the realizability gap.The teacher receives ground-truth information, while the student receives only the original problem.
HDPO: HDPO combines RL with privileged self-distillation targeted at cliff prompts where the RL gradient vanishes.The method augments standard RL rather than replacing it.
Theory: R = 1 filtered privileged generation recovers the optimal KL-regularized RL policy.The filter retains only correct privileged solutions, providing a theoretically justified teacher construction.
Results: HDPO improves pass@4 and pass@8 while maintaining greedy accuracy, with λ controlling the exploration–exploitation tradeoff.The result is demonstrated on OpenMathInstruct-2.

2 Background

RLVR uses binary outcome rewards, but GRPO receives no gradient on cliff prompts where every rollout fails. Existing solutions add machinery, whereas HDPO uses privileged generation and JSD distillation to learn from these prompts.

Background: RLVR formulates autoregressive generation as a sequential decision problem with a binary reward for solution correctness.The reward is assigned at completion rather than at individual reasoning steps.
Group Relative Policy Optimization: GRPO normalizes rewards within rollout groups and uses PPO-style ratio clipping, eliminating the need for a separate critic network.Training alternates between current-policy rollouts and clipped-objective updates.
The Cliff Problem: Cliff prompts are the all-failure case: every rollout receives zero reward, so the hardest problems receive essentially no gradient.Standard learning instead occurs when successes and failures provide contrast for advantage estimation.
Approaches to the Cliff Problem: Existing approaches address cliffs through curriculum filtering, hints, replay buffers, process rewards, or interleaved RL and distillation, adding substantial machinery.Filtering methods may skip cliff prompts rather than learning from them, while replay and process-reward methods introduce additional systems or models.
Approaches to the Cliff Problem: HDPO appends ground truth, generates privileged rollouts, filters correct solutions, and distills via JSD with a single unified mechanism.Its rationale is that the same model can solve a prompt when given privileged information even if it fails unprompted.
Knowledge Distillation for Reasoning: Knowledge distillation for reasoning must address distribution mismatch between teacher-generated sequences and the student’s inference-time distribution.Related self-distillation and unified KD-RL methods vary in how they combine teacher information, on-policy training, and reward optimization.
Knowledge Distillation for Reasoning: JSD is used to preserve broader solution coverage by providing mode-covering rather than purely mode-seeking distillation pressure.This connects HDPO’s distillation choice to prior work on diversity collapse in RLVR.

3 Method: Hybrid Distillation Policy Optimization

HDPO addresses cliff prompts by using the same model as a privileged teacher and unprivileged student, distilling only correct privileged trajectories. Its theory bounds the same-model gap and connects filtered generation to the KL-regularized RL optimum.

Core design: HDPO uses one model as both teacher and student, differing only because the teacher receives privileged ground-truth information.The shared weights remove the model-mismatch term present in cross-model distillation.
Core design: HDPO targets cliff prompts, where all standard rollouts fail and the GRPO policy gradient provides no learning signal.Privileged generation is restricted to these zero-reward prompts.
Training procedure: The distillation set retains only correct trajectories generated after injecting ground truth into cliff prompts, and applies token-level JSD distillation.The implementation approximates JSD with the teacher’s top-k=64 logits and exact tail correction.
Theoretical basis: Same-model privileged distillation has a tighter realizability bound than cross-model distillation because its divergence arises only from privileged information.A drifting teacher has zero model-mismatch term, whereas a frozen teacher introduces a non-zero term.
Theoretical basis: R = 1 rejection sampling from the reference policy recovers the KL-regularized RL optimum in the hard-threshold limit β →0+.For binary rewards, the optimizer converges to the reference distribution conditioned on correctness.
Theoretical basis: HDPO replaces the inaccessible correct-trajectory target on cliff prompts with a privileged proxy having non-empty support on correct solutions.The proxy remains approximate because privileged pass rates can be below one and sampling introduces noise.

4 Experiments

HDPO is evaluated against GRPO on OpenMathInstruct-2 using Qwen2.5-Math-1.5B-Instruct. It broadens sampled-solution coverage, while the distillation weight controls the tradeoff between coverage and greedy accuracy.

Setup: The evaluation uses 2048 validation problems from OpenMathInstruct-2, with Qwen2.5-Math-1.5B-Instruct trained for 2000 steps using GRPO.The comparison includes frozen and drifting teachers with λ ∈ {0.01, 0.1}.
Results: The drifting teacher at λ = 0.01 reaches pass@4 0.7861 (+1.1% over baseline) and pass@8 0.8271 (+0.4%), while pass@1 remains 0.6514 versus 0.6519.This configuration broadens coverage while essentially maintaining greedy accuracy.
Robustness: The λ = 0.01 improvement varies across hardware, although its direction is consistent and its magnitude is modest relative to run-to-run variance.This contrasts with the more robust λ = 0.1 pass@8 gains.
Results: At λ = 0.1, both teacher types achieve ~0.84 pass@8 (+1.4–1.7% over baseline), while pass@1 drops by ~2.3–2.8%.The larger coverage gains are more robust across hardware configurations.
Results: The drifting-teacher advantage is strongest at low λ and narrows at λ = 0.1 as the stronger distillation signal dominates.Thus, teacher choice matters less when distillation weight is high.

5 Discussion

HDPO broadens solution coverage through privileged distillation, with λ controlling the tradeoff between exploration and greedy accuracy. At λ=0.1, gains are larger but pass@1 declines, while λ=0.01 largely preserves greedy performance.

Coverage and tradeoffs: HDPO broadens mathematical solution coverage while λ controls how aggressively it trades greedy accuracy for pass@k.The mechanism supplies learning signal on cliff prompts where GRPO cannot.
Coverage and tradeoffs: At λ=0.01, coverage improves while pass@1 is essentially maintained, whereas λ=0.1 yields larger, more robust coverage gains at the cost of pass@1.The observed tradeoff is consistent at the 1.5B model scale.
Curriculum: The proposed expand-then-sharpen curriculum first broadens strategy support through distillation, then uses RL to restore a dominant mode and preserve secondary strategies.This hypothesis links mode-covering distillation with later mode-sharpening.
Teacher choice: The drifting teacher outperforms the frozen teacher at low λ, but their advantage narrows at λ=0.1 as distillation dominates.The frozen teacher at λ=0.1 achieves the highest reported pass@8, 0.8398, with a larger realizability gap.

6 Limitations

The empirical evidence is limited to one model scale and one dataset, while HDPO adds computation proportional to the number of cliff prompts. The authors propose delayed re-injection of formerly unsolvable prompts as a future expand-then-sharpen direction.

Scope: Empirical results cover only the 1.5B-parameter Qwen2.5-Math model on OpenMathInstruct-2, so improvement magnitudes may vary elsewhere.The authors specifically identify model capacity, dataset difficulty, and reward design as possible sources of variation.
Scope: Larger models may have fewer cliff prompts, potentially reducing HDPO’s marginal benefit, or may benefit more from expanded coverage.The direction of this scale effect remains unresolved.
Computational cost: HDPO adds privileged generation, filtering, top-k teacher-logit computation, and an extra JSD forward pass, with overhead proportional to cliff-prompt prevalence.Using the same vLLM instance for privileged and standard rollouts amortizes part of the cost.
Future direction: The proposed delayed re-injection cycle revisits formerly cliff prompts after intervening training to test whether distilled strategies are durable.The goal is to combine broad pass@k coverage with sharp pass@1 modes.

7 Conclusion

HDPO augments RL with privileged self-distillation to provide bounded, non-zero learning signals on cliff prompts. Theory supports tighter same-model realizability and optimal-policy recovery, while experiments show improved coverage with λ-controlled tradeoffs.

Conclusion: HDPO augments reinforcement learning with privileged self-distillation to address prompts where the standard RL gradient vanishes.The teacher receives ground-truth information while the student receives the original problem.
Conclusion: Same-model privileged distillation has a strictly tighter realizability gap than cross-model distillation because it avoids model mismatch.The remaining gap depends on the model’s Lipschitz constant and the ground-truth information content.
Conclusion: R=1 filtering recovers the optimal KL-regularized RL policy, providing theoretical justification for the privileged teacher construction.This result supports filtering privileged generations for correctness.
Conclusion: On OpenMathInstruct-2 with Qwen2.5-Math-1.5B-Instruct, HDPO consistently improves pass@4 and pass@8 while λ controls the exploration–exploitation tradeoff.At λ=0.1, pass@8 gains of +1.4–1.7% reproduce robustly across hardware; λ=0.01 gains are smaller and more variable.
Conclusion: The expand-then-sharpen paradigm could progressively reduce cliff prompts by broadening coverage with HDPO and sharpening modes with subsequent RL.The authors identify evaluation across larger models and diverse reasoning benchmarks as a next step.

A Realizability Gap Bound — Full Proof

The proof bounds same-model teacher–student divergence by the model’s Lipschitz behavior and the ground-truth-induced input change, then shows cross-model distillation adds model mismatch.

Same-model setup: The teacher and student use the same model weights but different contexts, with the teacher receiving ground truth and the student receiving only the prompt.The realizability gap is the KL divergence D_KL(P_T ∥ P_S).
Proof strategy: The proof rewrites KL divergence between softmax distributions as a centered cumulant-generating function and bounds it using bounded logit perturbations.The construction defines ψ(1)=D_KL(P ∥ Q) and controls the perturbation through its infinity norm.
Same-model bound: Under the Lipschitz assumption, the same-model realizability gap is controlled by L_θ multiplied by Δ(g), the input-space distance attributable to ground-truth tokens.The assumption applies on bounded contexts with finite embeddings and sequence length.
Interpretation: The bound depends on model behavior and ground-truth information content, not on a capacity gap between different models.Distillation difficulty scales with how much ground truth changes predictions rather than with architectural differences.
Cross-model comparison: Cross-model distillation decomposes divergence into model mismatch plus input perturbation, making its bound strictly larger than the same-model counterpart.The comparison remains strict even if the shared Lipschitz bound is quantitatively loose.
Caveat: The Lipschitz bound may be loose for deep transformers because layerwise composition can produce a large L_θ, but this affects both settings equally.The shared constant does not alter the comparison between same-model and cross-model distillation.

B Proposition 2 Full Proof

Proposition 2 shows that filtering privileged generations to correct trajectories recovers the RL-optimal policy in the hard-threshold limit for binary rewards.

Proposition statement: Proposition 2 establishes that R = 1 filtering yields the RL-optimal policy for binary rewards.The result is derived using the KL-regularized Gibbs distribution.
Binary-reward weighting: For binary reward R(τ) ∈ {0, 1}, the exponential weighting factor has only two possible values.Incorrect trajectories receive weight 1, while correct trajectories receive exp(1/β).
Binary-reward weighting: The partition function weights correct trajectories according to p = Pπ_ref(R = 1), the reference policy’s probability of generating a correct trajectory.This determines the relative weighting of correct and incorrect trajectories.
Hard-threshold limit: As β → 0+, exp(1/β) → ∞, so correct trajectories receive infinitely more weight than incorrect trajectories.The hard-threshold limit concentrates the distribution on trajectories with R(τ) = 1.
Hard-threshold limit: After dividing numerator and denominator by exp(1/β), the limiting expression follows because the incorrect-trajectory contribution vanishes.The relevant ratio tends to zero in the hard-threshold limit.

D Hardware Variation

The main results were obtained on 8×H200 GPUs and replicated on 8×H100 GPUs, with qualitative findings remaining consistent despite hardware-dependent numerical variation.

Replication setup: All five configurations were replicated on 8×H100 GPUs using identical hyperparameters and mathematically equivalent training computation.Differences arose from floating-point non-determinism across GPU microarchitectures.
H100 results: Table 3 reports best pass@k results on 8×H100 GPUs, with bold values indicating the best result in each column.It provides the table-based view of the H100 comparison.
Cross-hardware findings: λ = 0.1 improves pass@8 by +1.4–1.5% over baseline on both hardware configurations.This is the reported cross-hardware qualitative consistency.
Cross-hardware findings: Pass@1 remains highest for the baseline and λ = 0.01 configurations across the reported hardware comparisons.Greedy accuracy therefore does not favor the higher-coverage λ = 0.1 setting.
Hardware-sensitive result: On H200, drifting-0.01 achieves the highest pass@4 improvement at +1.1%, whereas on H100 the improvement is +0.4% and frozen-0.1 is best.The passage attributes this shift to differences near the noise floor.