Source-linked AI summary
Reinforcement Learning via Self-Distillation
Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, Andreas Krause
TL;DR
Existing RLVR methods use sparse scalar rewards, creating a credit-assignment bottleneck despite environments often exposing richer textual feedback. SDPO uses the current policy as a feedback-conditioned self-teacher for dense credit assignment, improving sample efficiency and final accuracy across reasoning, tool-use, and programming tasks, including standard RLVR settings and test-time discovery.
Problem
RLVR commonly learns from scalar outcome rewards even when environments provide feedback explaining why attempts failed, limiting credit assignment.
Method
SDPO samples on-policy rollouts, conditions the current policy on received feedback, and distills the resulting next-token predictions back into the policy.
Results
SDPO improves sample efficiency and final accuracy over strong RLVR baselines across scientific reasoning, tool use, and LiveCodeBench v6, and accelerates test-time solution discovery.
Takeaways & Limitations
SDPO converts tokenized feedback into self-supervision and can also use successful rollouts as implicit feedback in scalar-reward environments.
Takeaways & Limitations
SDPO depends on in-context learning and informative feedback, may underperform GRPO on weaker models, and adds retrospective log-probability computation overhead.
Abstract
from arXiv · showhide
Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiable environments actually provide rich textual feedback, such as runtime errors or judge evaluations, that explain why an attempt failed. We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO), which converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. In this way, SDPO leverages the model's ability to retrospectively identify its own mistakes in-context. Across scientific reasoning, tool use, and competitive programming on LiveCodeBench v6, SDPO improves sample efficiency and final accuracy over strong RLVR baselines. Notably, SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts. Finally, applying SDPO to individual questions at test time accelerates discovery on difficult binary-reward tasks, achieving the same discovery probability as best-of-k sampling or multi-turn conversations with 3x fewer attempts.
1 Introduction
RL post-training in verifiable domains is limited by scalar rewards that obscure why attempts fail. SDPO uses feedback-conditioned self-distillation to provide dense credit assignment and improves efficiency and accuracy across evaluated settings.
- Motivation: Scalar RLVR rewards create a credit-assignment bottleneck by revealing outcomes without explaining failures.When all rollouts receive the same reward, GRPO advantages can collapse to zero and learning can stall.
- Motivation: RLRF represents environment feedback as tokens, combining reward information with observations such as runtime errors, failing tests, or judge evaluations.This exposes more state information than a scalar reward alone.
- Method: SDPO conditions the current policy on feedback and distills its feedback-informed next-token predictions into the policy as dense, logit-level supervision.The current model acts as a self-teacher that retrospectively identifies mistakes without requiring an external teacher.
- Results: 70.2% vs. 66.6% final accuracy shows SDPO outperforming an improved GRPO baseline on aggregate in standard scalar-reward environments.SDPO also achieves higher accuracy with up to 11× shorter generation lengths.
- Results: 48.8% vs. 41.2% final accuracy shows SDPO outperforming GRPO on LiveCodeBench v6, while reaching GRPO’s final accuracy in 4× fewer generations.The comparison uses Qwen3-8B and LeetCode-style feedback.
2 SDPO: Self-Distillation Policy Optimization
The paper frames SDPO as an algorithm that uses the current policy’s in-context learning ability to assign credit.
- SDPO uses the current policy’s in-context learning ability for assigning credit.
- The method’s key object is the feedback-conditioned self-teacher πθ(· | x, f).
- The self-teacher is defined as the model conditioned on the question and feedback.
1. Question
The section asks for a Python function that returns all numbers from 1 to n and requests a brief answer.
- The requested task is to write a Python function.
- The function should return all numbers from 1 to n.
- The answer should be brief.
3. Feedback
The provided answer defines a Python function that returns the integers from 1 through n as a list.
- The answer defines the function numbers_up_to_n(n).
- The function returns list(range(1, n + 1)).
- The implementation produces numbers from 1 through n inclusively.
4. Credit assignment by self-teacher
SDPO uses the current policy as a feedback-conditioned self-teacher to convert rich feedback into dense, token-level credit assignment. It re-evaluates original attempts under feedback and distills those predictions into the student policy.
- Self-distillation policy optimization: SDPO repeatedly distills feedback-conditioned self-teacher predictions into the student using a logit-level distillation loss.The teacher and student share the same policy parameters, while gradients are blocked through the teacher.
- Self-teacher mechanism: The self-teacher re-evaluates the original attempt after seeing feedback, identifying where the student was wrong without requiring an explicit solution.The feedback-conditioned model can retrospectively adjust token predictions for the original response.
- Comparison to RLVR: Unlike GRPO’s rollout-level advantages, SDPO’s advantages vary across tokens and are zero only where student and teacher agree.This enables credit assignment at the locations where feedback-conditioned predictions differ from the original policy.
- Comparison to RLVR: SDPO extends scalar RLVR feedback to arbitrary token sequences and uses them to estimate dense logit-level advantages.Tokens more likely under the teacher receive positive advantages, while tokens less likely under it receive negative advantages.
- Compute time & memory: SDPO adds self-teacher log-probability computation, which can be parallelized and is substantially faster than sequential generation.Top-K distillation with K = 100 avoids virtually any memory overhead while retaining most of the information.
3 Learning without Rich Environment Feedback
In standard RLVR environments, SDPO uses successful attempts as implicit feedback for failed attempts and outperforms a strong GRPO baseline across scientific reasoning and tool-use tasks. It also reaches higher accuracy with substantially shorter generations and faster training.
- Learning without Rich Environment Feedback: SDPO treats successful attempts in the current batch as feedback for failed attempts on the same question.This lets the self-teacher compare an incorrect attempt with a correct solution and provide dense credit assignment.
- Results: SDPO outperforms GRPO across almost all runs and reaches the 5h GRPO accuracy after 1 hour of training in several cases.On Chemistry with Olmo3-7B-Instruct, SDPO reaches the 5h GRPO accuracy in 50 minutes, a 6× speedup.
- Self-distillation learns to reason concisely: SDPO produces more than 3× shorter generations than GRPO across tasks while achieving higher accuracy.On Chemistry with Olmo3-7B-Instruct, SDPO reduces response length by 11× relative to GRPO while maintaining higher accuracy.
- Self-distillation learns to reason concisely: GRPO’s longer responses often contain filler phrases and circular repetitions, whereas SDPO’s generations remain concise and avoid these patterns.The paper links this qualitative difference to SDPO’s dense credit assignment.
4 Learning with Rich Environment Feedback
SDPO improves coding performance and training efficiency by using rich feedback for dense credit assignment, with gains that are stronger for larger models and a self-teacher that improves during training.
- Coding results: 48.8% final LCBv6 accuracy for SDPO exceeds GRPO’s 41.2% and reaches GRPO’s final accuracy in 4× fewer generations.SDPO also exceeds Claude Sonnet 4 at 40.5% and Claude Opus 4 at 39.7% on the public LCBv6 leaderboard.
- Scaling: SDPO significantly outperforms GRPO on larger Qwen3 models but only slightly improves over GRPO on smaller models.The results link SDPO’s benefit to the base model’s in-context learning and retrospection ability.
- Credit assignment: Logit-level SDPO yields significant gains over token-level and sequence-level SDPO, while sequence-level SDPO still outperforms GRPO.
- Self-teacher dynamics: The self-teacher improves significantly during training, allowing the student to surpass the initial teacher’s accuracy and bootstrap from a weak model to a stronger one.
- Feedback design: Environment output and sample solutions provide complementary feedback, while including the student’s original attempt biases the teacher and reduces exploration.
5 Solving Hard Questions via Test-Time Self-Distillation
Test-time SDPO repeatedly uses feedback to compress interaction history into model weights, accelerating solution discovery on hard binary-reward coding questions beyond best-of-k and multi-turn sampling.
- Method: SDPO updates model weights from feedback after each attempt instead of retaining the full interaction history in the context window.
- Evaluation: The evaluation includes 19 hard questions with pass@64 < 0.5 and 9 very hard questions with pass@64 < 0.03.
- Results: 53.2% of very hard questions reach discovery@2750 with SDPO, versus 41.5% for best-of-k and 35.6% for multi-turn sampling.
- Results: SDPO reaches a 22% discovery probability on very hard questions with approximately 3× fewer generations than best-of-k and multi-turn sampling.
- Results: SDPO reaches 78% discovery@2750 on hard questions and achieves 67% discovery with roughly 2.4× fewer generations than best-of-k and multi-turn sampling.
- Results: SDPO uniquely solves Question 3 after 321 attempts, whereas best-of-k and multi-turn sampling do not solve it within 2750 attempts.This corresponds to 20 self-distillation iteration steps with batch size 16.
- Self-teacher dynamics: The initial self-teacher solves almost none of the hard questions, yet its credit assignment enables iterative refinement that eventually solves them.
6 Related Work
Related work spans scalar-reward RLVR, process reward models, feedback-guided generation, feedback-conditioned policies, and self-distillation, while SDPO combines on-policy learning with feedback-based self-distillation.
- RL and credit assignment: RLVR methods commonly use outcome rewards, while process reward models estimate intermediate rewards to improve credit assignment.
- Rich feedback: Several approaches use execution or verbal feedback by mapping it to rewards, often with external frozen models or strong external LLMs.
- Feedback-based learning: Other methods use feedback for in-context improvement or manually curate before-and-after preference data for direct preference optimization.
- Feedback conditioning: Feedback-conditioned policies treat feedback as a goal, whereas RLRF treats feedback as a state for determining the next action.
- Self-distillation: Self-distillation transfers behavior or information from a context-augmented model into the model’s weights, extending beyond standard supervised settings.
- Self-distillation: SDPO differs from off-policy self-distillation by training on the student’s own generations and using on-policy feedback-based learning.
7 Conclusion, Limitations, and Future Work
The paper introduces RLRF and SDPO, showing that feedback-conditioned self-distillation provides dense credit assignment and improves efficiency across training and test-time settings. It also identifies dependence on model in-context learning, feedback quality, and computational overhead as limitations, while outlining broader evaluation directions.
- Conclusion: SDPO uses the current policy as a feedback-conditioned self-teacher and distills corrected log-probabilities into the student for dense credit assignment.The method is presented as a minimal modification to standard RLVR pipelines.
- Conclusion: SDPO improves sample efficiency and wall-clock convergence over GRPO on reasoning tasks, including standard RLVR environments without rich feedback.The paper reports that gains grow with model scale and that test-time SDPO accelerates solution discovery on difficult binary-reward tasks.
- Conclusion: SDPO converts diverse tokenized feedback into effective self-supervision by retrospectively determining how the model should have acted.The paper characterizes this as utilizing precise outcomes rather than only binary rewards.
- Limitations: SDPO depends on in-context learning ability and feedback quality, may underperform GRPO on weaker models, and adds computational overhead for retrospective log-probability computation.The overhead is often negligible but can matter more for smaller models with shorter generation lengths.
- Future Work: Future work includes evaluating SDPO in long-horizon agentic settings, scaling it to frontier models, and testing it beyond verifiable rewards.The paper also proposes studying behavioral differences in reasoning and the effects of reprompt templates.
A Implementation of SDPO
The implementation integrates SDPO into a standard RL pipeline by reprompting batches with teacher context and applying per-token logit divergences. Practical choices address gradient estimation, teacher stabilization, memory use, and off-policy corrections.
- Pipeline: SDPO modifies a standard RL training pipeline by reprompting batches with teacher context and applying a per-token divergence.The divergence may be reverse-KL, forward-KL, or Jensen-Shannon.
- Teacher: The self-teacher is defined as qθ(· | x, f) := πθ(· | reprompt(x, f)), where reprompt specifies the teacher’s feedback-conditioned template.The teacher and student therefore share the model parameters but use different contexts.
- Gradient Estimators: SDPO supports per-token and sequence-level gradient estimators for the KL divergence between student and teacher policies.The sequence-level estimator accounts for how token choices influence future prefixes, but experiments found no measurable gains relative to its added complexity.
- Teacher Regularization: Teacher stabilization uses either an explicit trust region or an exponential moving average, trading memory usage against additional log-probability computation.EMA requires extra GPU memory but no runtime overhead, whereas the trust-region approach avoids that memory when using the reference policy for regularization.
- Efficiency: Top-K student-token distillation reduces memory requirements without significantly affecting performance because most vocabulary tokens are uninformative at each step.The implementation also generalizes PPO-style objectives to logit-level losses with per-logit advantages and uses token-level importance sampling in experiments.
B Theoretical Analysis
The theoretical analysis relates SDPO to policy-gradient estimators, trust-region teacher updates, maximum-entropy reinforcement learning, and implicit reward construction. It also situates the method among prior approaches to credit assignment and test-time training.
- Teacher Regularization: A trust-region teacher stays close to the target teacher while remaining constrained relative to the initial reference teacher.The solution is characterized as minimizing cross-entropy to qθ while maximizing cross-entropy to qθref under the constraint.
- Related Work: Prior credit-assignment methods include value networks and external reward models, while verifiable LLM environments can be interpreted as partially observable because feedback reveals latent states.The paper also relates test-time self-distillation to test-time training.
- Maximum-Entropy RL: SDPO can be viewed as maximum-entropy RL with dense implicit rewards r(yt | x, y<t) = log qθ(yt | x, f, y<t).The retrospective model supplies the implicit reward used in the equivalent maximum-entropy objective.
- Connections: The paper connects SDPO to inverse reinforcement learning because the student learns an implicit reward function defined by the retrospective model.This connection concerns recovering a reward-like signal from the teacher’s behavior rather than using an explicit reward model.
D.2.1 Additional Results
Additional experiments examine SDPO across question difficulty, training configurations, model sizes, logged dynamics, baselines, and test-time discovery. They report stronger gains on medium and hard coding problems and faster discovery than comparison methods.
- Difficulty: SDPO significantly improves over GRPO on medium and hard LCBv6 questions, highlighting the value of rich feedback for challenging tasks.Figure 15 stratifies average accuracy by easy, medium, and hard questions.
- Additional Evaluations: The additional analyses include results for Qwen2.5-Instruct, training loss and dynamics, and comparisons with GSPO and CISPO.Logged metrics include entropy, average gradient norm, and average response length.
- Evaluation Caveat: One malformed question was discarded because rounding inaccuracies caused failures even when the underlying logic was correct.This exclusion affected the selection of hard questions.
- Test-Time Discovery: SDPO reaches successes faster than best-of-k sampling and multi-turn sampling, with up to 13.6× speedup on individual questions.Mean generations are truncated at the maximum budget of 2750, and very hard tasks are marked separately.
- Test-Time Discovery: The retrospective teacher’s initial accuracy is near or exactly 0% for most hard and very hard tasks, yet self-distilled token-level advantages still support iterative refinement.The analysis includes per-question teacher accuracy and discovery curves for hard questions.
E Experiment Details
Experiments use a multi-GPU implementation with documented SDPO and GRPO hyperparameters, task-specific prompts, and illustrative tool-use and coding examples. The section also presents an advantage visualization comparing SDPO with GRPO and details the coding-task setup and solution strategy.
- Infrastructure: Experiments run on a single node with four NVIDIA GH200 GPUs totaling 378GB VRAM, using CUDA 12.8 and PyTorch 2.7.0.
- Infrastructure: The implementation uses verl, FSDP2 for distributed training, and vLLM for efficient batched rollout inference.
- Hyperparameters: SDPO and GRPO hyperparameters are summarized in Tables 12 and 13, with configurations selected by highest validation accuracy after grid searches.The searches vary learning rates and minibatch sizes; SDPO additionally varies forward KL versus Jensen–Shannon KL.
- Prompting: Multiple-choice and tool-use experiments use task-specific prompt templates, including structured tool descriptions, parameters, outputs, and action formatting.
- Advantage visualization: Figure 21 compares SDPO and GRPO advantages, with rows representing response beginnings and colors indicating positive or negative token-level advantage values.Positive advantages are blue and negative advantages are red.
- Competitive programming setup: The coding example defines a binary-string task with at most one trade, where a surrounded block of ones is changed to zeros before a surrounded block of zeros is changed to ones.The augmented boundary ones do not contribute to the final count, and the described strategy evaluates candidate trades to maximize active sections.