Source-linked AI summary
H$^2$SD: Hybrid Hindsight Self-Distillation
Qiye Cai, Yichuan Ma, Peiji Li, Yongkang Chen, Qipeng Guo, Yicheng Zou, Linyang Li, Xiaocheng Feng, Bing Qin
TL;DR
RLVR rewards entire trajectories, limiting token-level credit assignment, while existing self-distillation methods trade corrective guidance against stable optimization. H2SD routes successful and failed trajectories through different teacher contexts and update rules, achieving the strongest aggregate performance across the evaluated reasoning benchmarks.
Problem
RLVR supplies a scalar trajectory reward, leaving limited information about which tokens caused reasoning success or failure.
Method
H2SD adapts teacher context and updates to correctness, refining successful trajectories with rephrased-response probabilities and correcting failed ones via reverse-KL reference hints.
Results
H2SD achieves the strongest aggregate performance across Sudoku, Calcudoku, and Arrow Maze among the evaluated methods.
Takeaways & Limitations
Trajectory correctness can guide both teacher context and update strategy for stable credit refinement and corrective reasoning supervision.
Takeaways & Limitations
The comparison highlights a trade-off: direct teacher matching risks privileged-information leakage, while magnitude-only modulation weakens distribution learning.
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning, but a scalar trajectory reward offers limited token-level guidance. Existing self-distillation methods add a privileged teacher but typically assign it a fixed role: direct distribution matching may destabilize successful behavior, while magnitude-only modulation offers little corrective guidance after failure. We observe that successful and failed trajectories require different forms of hindsight supervision. A successful response already contains a valid student-generated reasoning path and can therefore serve as privileged context rather than being replaced by an external rationale. A failed response, however, requires corrective reference information. We introduce Hybrid Hindsight Self-Distillation ($\mathrm{H}^{2}\mathrm{SD}$), which jointly adapts teacher context and update strategy to trajectory correctness. For successful trajectories, we construct the teacher context from the verified response and a rephrasing instruction, and use the teacher only to re-evaluate the original response tokens. The resulting probabilities refine token credit assignment without changing the direction determined by the reward. For failed trajectories, a verified reference hint provides corrective guidance through reverse-KL distillation. Experiments on challenging reasoning benchmarks show that H$^2$SD achieves the strongest overall performance among representative RLVR and self-distillation baselines, with stable optimization and a favorable accuracy-efficiency trade-off.
1. Introduction
RLVR’s scalar outcome rewards limit token-level credit assignment, while existing self-distillation approaches trade corrective guidance against optimization stability. H2SD addresses this trade-off by adapting teacher context and supervision strategy to trajectory correctness, using successful responses for stable credit refinement and reference hints for failed-trajectory correction.
- Motivation: RLVR’s single scalar outcome reward provides limited information about which response tokens caused success or failure, creating a credit-assignment bottleneck.On-policy distillation is introduced as a way to provide denser supervision over the response.
- Motivation: Existing self-distillation methods face a trade-off between distribution-level corrective guidance and stable token-level credit assignment.Direct teacher-distribution matching can destabilize training, whereas update-magnitude modulation offers weaker corrective supervision.
- H2SD: H2SD adapts the teacher’s role to trajectory correctness: successful trajectories use rephrased confirmed responses to modulate updates, while failed trajectories use reference hints with reverse KL correction.The successful-trajectory teacher evaluates the original response tokens, preserving the reward-determined optimization direction; failed trajectories receive corrective reference information.
- H2SD: H2SD uses stronger-language-model-generated reference hints containing key intermediate reasoning steps and verifier-confirmed final answers, with shared hints controlling comparisons among hint-based methods.OPSD, RLSD+hint, SRPO+hint, and H2SD use the same hint set whenever external hints are used.
- Results: H2SD achieves the best overall performance among evaluated methods on Sudoku, Calcudoku, and Arrow Maze, including GRPO, OPSD, RLSD, SDPO, and SRPO.The reported design refines credit assignment on successful trajectories and corrects failed trajectories using teacher distributions conditioned on reference hints.
2. Related Work
Related work progresses from outcome-based RLVR and on-policy distillation toward self-distillation with privileged information and more stable reward-grounded updates. These methods improve token-level supervision or reduce external-teacher dependence, but introduce challenges involving teacher reliance, information leakage, and training stability.
- Reinforcement Learning with Verifiable Rewards: RLVR optimizes language-model reasoning with automatically verifiable outcome rewards, while GRPO, DAPO, and GSPO improve advantage estimation, stability, efficiency, or credit assignment.GRPO estimates group-relative advantages without an explicit value model.
- On-Policy Distillation: On-policy distillation uses student-generated trajectories and teacher evaluation to provide dense token-level supervision while mitigating off-policy distribution shift.Its benefits combine on-policy exploration with dense supervision, though it typically requires an additional strong teacher.
- On-Policy Self-Distillation with Privileged Information: Privileged-information self-distillation uses the same model or auxiliary revisers to guide students with ground-truth solutions, refined responses, and richer contextual signals.OPSD uses token-level distribution matching, while OPCD and GATES extend privileged information beyond final answers.
- Stable Self-Distillation for RLVR: Stable RLVR self-distillation lets environment rewards determine update direction while privileged teachers modulate token-level magnitudes, with related work adding confidence gating, evidence reweighting, or outcome-based routing.RLSD addresses information leakage and training instability; the supplied passage truncates SRPO’s routing description after successful and failed trajectories.
3. Preliminaries
This section reviews GRPO and successive distillation methods that provide increasingly fine-grained token supervision. It highlights OPSD’s leakage risk and RLSD’s limited distribution learning, especially for failed responses.
- GRPO: GRPO assigns every token in a response the same reward or penalty magnitude from the response-level relative advantage and optimizes a clipped surrogate objective.Its objective also includes a KL penalty against a reference policy.
- OPD: OPD uses a stronger, vocabulary-compatible teacher to provide next-token distributions and fine-grained supervision instead of GRPO’s trajectory-level scalar reward.The teacher supplies diverse training signals within the same trajectory.
- OPSD: OPSD avoids OPD’s external-teacher and vocabulary-consistency requirements by using the same model as teacher and student, with privileged information conditioning the teacher.This preserves differentiated token-level signals and alleviates scalar-reward sparsity.
- RLSD: RLSD reframes OPSD distribution matching as token-level credit assignment, redistributing reward or penalty strength according to privileged-information support while preserving the trajectory-advantage sign.It clips token weights to limit individual-token influence and stabilize training.
- RLSD: RLSD reduces information leakage but does not learn the teacher’s full output distribution, losing preferences over reasoning expressions and relations among candidate tokens.This weakness is most evident for failed responses, where magnitude modulation provides no explicit correction direction.
4. H2SD: Hybrid Hindsight Self-Distillation
H2SD uses a stronger LLM to generate training-only natural-language hints while retaining the current model as the self-teacher. It adapts distillation to trajectory correctness, refining credit assignment for successful responses and providing corrective guidance for failed ones.
- Hint generation: A stronger LLM generates natural-language hints that provide useful problem-solving information as privileged training-time context.The stronger model generates hints rather than directly serving as the teacher.
- Successful trajectories: For successful trajectories, H2SD conditions the teacher on the verified response and a rephrasing instruction, then re-evaluates the original response tokens.The resulting probabilities compute RLSD magnitude weights while preserving the positive reward’s optimization direction.
- Overall objective: H2SD selects its distillation strategy according to trajectory correctness, combining RLSD-based credit refinement for successes with corrective distillation for failures.The corrective-distillation strength is controlled by γ.
- Failed trajectories: For failed trajectories, H2SD uses a hint-conditioned teacher distribution as corrective guidance and minimizes reverse KL divergence from student to teacher.Negative rewards discourage sampled actions but provide limited information about how to correct them.
5. Experiments
Across four procedurally generated reasoning benchmarks, H2SD delivers the strongest overall performance and best accuracy–generation-cost trade-off. Ablations attribute its gains to outcome-conditioned routing, process-level hints, and magnitude modulation for successful trajectories rather than direct reverse-KL matching.
- Main results: H2SD achieves the strongest overall performance across four benchmarks, with especially pronounced advantages on Sudoku.Methods using identical privileged hints nonetheless differ in performance, indicating that converting information into learning signals is the primary source of gains.
- Main results: H2SD attains the highest average Calcudoku performance and ranks first across all Arrow Maze sizes, including held-out 8×8 and 10×10 settings.It consistently outperforms SRPO and remains substantially stronger than SRPO+hint overall.
- Efficiency: H2SD achieves the highest accuracy with the lowest average generation cost on Sudoku-6×6, showing that its gains do not depend on longer responses.Figure 3 characterizes this as the most favorable accuracy–efficiency trade-off among the compared methods.
- Routing ablation: Outcome-conditioned routing improves accuracy: H2SD outperforms uniform strategies, whereas Reversed Routing falls below the Base LLM in every setting and rapidly collapses actor entropy.The results support using teacher magnitude modulation for successful trajectories and directional correction for failed trajectories.
- Privileged contexts: The hint performs best across evaluated privileged contexts, reaching 76.50%, 57.25%, and 24.20% on Sudoku 6×6, Sudoku 8×8, and Arrow Maze, respectively.Sibling solutions reach 61.75%, 48.50%, and 19.60% on the same settings, while ground-truth answers and programmatic feedback provide only limited improvements.
- Update-strategy ablation: Rephrasing improves Magnitude Only by 23.75 and 32.00 percentage points on Sudoku-6×6 and Sudoku-8×8, while improving H2SD by 8.00 and 5.75 points.Rephrasing sharply reduces entropy under direct Reverse-KL Only, whereas Magnitude Only remains relatively stable and H2SD largely preserves stability.
6. Conclusion
H2SD selects both teacher context and update rule based on trajectory correctness. It refines token-level credit assignment for verified trajectories and provides corrective guidance for failed ones, achieving the strongest aggregate performance across the evaluated tasks.
- Conclusion: H2SD uses trajectory correctness to select both the teacher context and update rule.The framework adapts its self-distillation strategy to whether a trajectory is verified or failed.
- Conclusion: For verified trajectories, H2SD refines credit assignment across tokens, while failed trajectories receive corrective guidance from reference hints.The two cases use distinct forms of hindsight supervision.
- Conclusion: H2SD achieves the strongest aggregate performance among evaluated methods across Sudoku, Calcudoku, and Arrow Maze.The conclusion reports this result across all three named benchmarks.
A. Implementation Details · A.1. Technical Setup
Experiments use Qwen3-30B-A3B-Instruct-2507 with a shared VERL/SGLang implementation and full-parameter FSDP training on NVIDIA H200 GPUs. Evaluation reports pass@1 accuracy from a single greedily decoded response accepted by the task-specific verifier, while teacher-update policies vary by method.
- A.1. Technical Setup: All experiments use Qwen3-30B-A3B-Instruct-2507 as the base model.
- A.1. Technical Setup: All methods are implemented in VERL.
- A.1. Technical Setup: SGLang serves as the rollout engine for the experiments.
- A.1. Technical Setup: Training updates all parameters using Fully Sharded Data Parallel on NVIDIA H200 GPUs.
- A.1. Technical Setup: Pass@1 accuracy is the fraction of validation problems whose single greedily decoded response is accepted by the task-specific verifier.
- A.1. Technical Setup: RLSD, RLSD+Hint, OPSD, and H2SD use frozen teacher copies initialized from the initial policy, without student synchronization.
- A.1. Technical Setup: SDPO, SRPO, and SRPO+Hint instead use their respective EMA teacher updates, while GJSD and IS denote generalized Jensen–Shannon divergence and importance sampling.
A.2. Method-Specific Hyperparameters … C. Prompt Templates
The supplementary material specifies method-specific hyperparameters, offline hint generation and verification, representative task hints, and prompt templates for failed and successful trajectories. Successful-trajectory prompts preserve the verified answer while training the teacher to score the original response tokens.
- A.2. Method-Specific Hyperparameters: Method-specific hyperparameters are reported separately for GRPO, SDPO, SRPO, RLSD, RLSD+Hint, OPSD, and H2SD.SDPO omits distillation targets without a verified-correct sibling, while SRPO uses a policy-gradient fallback in that case.
- A.3. Offline Hint Generation: Reference hints for RLSD+Hint, OPSD, and failed-trajectory H2SD are generated offline with Kimi-K2.6 and are not used during policy optimization.The generation configuration is summarized separately in Table 12.
- B. Representative Hint Examples: For each training problem, Kimi K2.6 produces a natural-language hint containing key intermediate deductions and a final answer.A hint is retained only when its final answer is accepted by the task verifier.
- B. Representative Hint Examples: Intermediate hint deductions are not independently verified, and hints are shown only to the self-teacher during training, not to the student at inference.Tables 13–15 provide representative verifier-accepted hints for Sudoku-6 × 6, Calcudoku-5 × 5, and Arrow Maze-6 × 6.
- C.1. Hint Generation: Failed-trajectory privileged contexts use Kimi-K2.6-generated reference rationales containing the solution process and final answer inside <hint>...</hint> tags.The Sudoku-6×6 template instructs the model to solve the puzzle from the beginning, summarize the reasoning, and provide the answer in the tags.
- C.1. Hint Generation: The failed-trajectory hint-generation prompt receives only the original problem, represented by {prompt}.The prompt template is illustrated using a complete Sudoku problem.
- C.2. Rephrasing Instruction for Successful Trajectories: For successful trajectories, the self-teacher receives the verifier-confirmed student response and a rephrasing instruction but scores the original tokens instead of generating a rewritten response.The instruction preserves the Sudoku-solving flow and copies the final <answer>...</answer> block verbatim, while teacher forcing scores the original trajectory.
- C.2. Rephrasing Instruction for Successful Trajectories: The successful-trajectory template uses {prompt} for the problem and {first_response} for the original verified trajectory.It requires the output to retain a solution process and end with the unchanged answer block.
D. Actor Entropy and Exploration
Actor entropy remains stable under Magnitude Only and H2SD, while Reverse-KL Only reduces exploration and Reversed Routing rapidly collapses it. Rephrasing preserves this stability in H2SD but sharply lowers entropy when combined with direct reverse-KL distillation.
- Effect of update routing: Magnitude Only and H2SD maintain relatively stable actor entropy, preserving policy diversity throughout training.Actor entropy measures next-token uncertainty and serves as a diagnostic proxy for exploration and output diversity.
- Effect of update routing: Reverse-KL Only reduces actor entropy, while Reversed Routing rapidly drives it toward zero and collapses exploration.Reversed Routing applies reverse-KL distillation to successful trajectories and magnitude modulation to failed trajectories.
- Interaction with the rephrasing instruction: Under Magnitude Only, rephrasing maintains comparable or slightly higher actor entropy while refining successful reasoning without changing the policy-gradient direction.The teacher changes response-token update magnitudes rather than replacing the policy-gradient direction.
- Interaction with the rephrasing instruction: Under Reverse-KL Only, rephrasing sharply decreases actor entropy on every task, imposing concentration pressure that limits continued exploration.Runs without rephrasing remain more stable, whereas direct fitting to the rephrased teacher distribution can reduce policy diversity.
- Interaction with the rephrasing instruction: H2SD keeps entropy curves with and without rephrasing closely aligned and stable by using rephrased teacher probabilities for successful-trajectory magnitude modulation and reverse-KL only for failed trajectories.This improves token-level credit assignment without the broad entropy reduction caused by uniformly combining rephrasing with reverse-KL updates.
E. Detailed Analysis of Generation Efficiency
Across four evaluation settings, H2SD delivers a favorable accuracy–generation-cost trade-off and lies on or near the Pareto frontier. Its efficiency reflects more successful and concise reasoning trajectories, not merely shorter outputs.
- Accuracy–efficiency trade-off: H2SD consistently lies on or near the Pareto frontier across four settings, showing that its performance gains are not attributable to a larger generation budget.Response length and accuracy do not follow a simple monotonic relationship across methods.
- Accuracy–efficiency trade-off: H2SD achieves the strongest aggregate accuracy across all four settings while producing shorter responses than most competing methods.SDPO sometimes generates shorter outputs, but its substantially lower accuracy shows that brevity alone does not indicate reasoning quality.
- Trajectory-length decomposition: Incorrect responses are substantially longer than correct responses for every evaluated method, so efficiency depends on both trajectory success rates and unnecessary continuation.The Calcudoku analysis decomposes average response length by prediction correctness.
- Trajectory-length decomposition: H2SD benefits from higher accuracy, among-the-shortest correct responses, and failed responses considerably shorter than those of several reward-based and magnitude-modulation baselines.These effects reduce the contribution of long failed trajectories and limit unnecessary continuation after incorrect reasoning.
- Interpretation: H2SD improves reasoning-trajectory composition by reaching valid solutions more frequently, maintaining concise successful trajectories, and limiting continuation after incorrect reasoning.Because response length is only a proxy for inference cost, these findings support improved generation efficiency rather than a direct causal benefit from shorter reasoning.
F. Limitations
H^2SD's unsuccessful-trajectory supervision depends on verifier-confirmed natural-language hints generated offline by an external model. Although that model is absent from policy optimization and inference, the pipeline still requires access to a sufficiently capable model.
- External hint generation: H^2SD constructs privileged context for unsuccessful trajectories from verifier-confirmed natural-language hints.Only hints whose final answers pass the task-specific verifier are retained.
- External hint generation: In the experiments, Kimi-K2.6 generates the hints offline, and it supplies neither distillation logits nor online supervision.The external model is not queried during policy optimization or inference.
- External hint generation: The current pipeline still depends on access to a sufficiently capable external model.This dependency remains despite offline hint generation and the model's absence during optimization and inference.