Reinforcement Learning via Self-Distillation

Evaluation results

• 48.8% vs. 41.2% final accuracy on LiveCodeBench v6 shows SDPO substantially outperforming GRPO, while matching GRPO’s final accuracy in 4× fewer generations.SDPO’s gains grow with model scale, suggesting stronger in-context learning improves self-teaching.Source: Figure 1 · p. 1, §1 · p. 3

Motivation

• Scalar outcome rewards create a credit-assignment bottleneck because binary or shared rewards hide why attempts fail and can make GRPO advantages collapse to zero.External teachers can provide dense supervision, but are often unavailable when online learning aims to exceed existing models.Source: §1 · p. 2

Method

• SDPO conditions the current policy on tokenized feedback, using its altered next-token distribution as a self-teacher that supplies dense logit-level credit assignment without external supervision or sampling overhead.The on-policy algorithm minimizes a logit-level distillation loss, and its self-teacher-derived advantages require only minor changes to standard RLVR pipelines.Source: §1 · pp. 2–3

Evaluation results

• 70.2% vs. 66.6% aggregate final accuracy and up to 11× shorter generations show SDPO outperforming an improved GRPO baseline in scalar-reward RLVR.SDPO treats successful attempts in the current batch as feedback for failed attempts on the same question, across scientific reasoning and tool-use tasks.Source: §1 · p. 3
• 3× faster solution discovery on difficult binary-reward LiveCodeBench questions demonstrates that Test-Time Self-Distillation accelerates learning before the first solution is found.The evaluated base model has pass@64 below 0.03, and SDPO specializes to individual test questions through test-time training.Source: §1 · p. 3

2 SDPO: Self-Distillation Policy Optimization

• SDPO proposes using the current policy’s in-context learning ability for credit assignment.Source: §2 · p. 3
• The method’s key object is a self-teacher derived from the current policy.Source: §2 · p. 3
• The self-teacher is represented as πθ(· | x, f ).Source: §2 · p. 3

Question

• Implement a Python function for the requested task.The function should return all numbers from 1 to n.Source: §1 · p. 4
• The function’s output must include all numbers from 1 to n.Source: §1 · p. 4
• Provide the answer briefly.Source: §1 · p. 4

• The Python function numbers_up_to_n(n) returns list(range(1, n + 1)), producing integers from 1 through n.Source: §3 · p. 4

4. Credit assignment by self-teacher

• SDPO distills feedback-conditioned self-teacher predictions into the student, using retrospection to identify erroneous tokens and assign credit sparsely.The teacher is stop-gradient and conditions on environment output or a previously successful solution; Qwen3-8B can identify errors without an explicit solution.Source: §4 · p. 4, Figure 4

2.1 Comparison to RLVR

• SDPO extends RLVR from 1-bit feedback to arbitrary token sequences and replaces rollout-level advantages with dense logit-level advantages.Advantages are positive where tokens are more likely under the teacher and negative where they are less likely, while matching tokens receive zero advantage.Source: §2.1 · p. 5

2.2 Compute time & memory

• SDPO adds only parallelizable self-teacher log-probability computation, while top-K distillation with K = 100 avoids virtually any memory overhead.The method can reuse standard RLVR implementations and extend to off-policy data with PPO-style clipped importance sampling.Source: §2.2 · pp. 5–6, §2.1 · p. 5

2.3 Stability improvements

• Training stability improves with a regularized self-teacher using EMA or teacher interpolation and with the symmetric Jensen-Shannon divergence.Both teacher regularization strategies are reported to stabilize learning.Source: §2.3 · p. 6

3 Learning without Rich Environment Feedback

• SDPO treats successful attempts in the current batch as feedback for failed attempts, allowing the self-teacher to identify errors and provide dense credit assignment without modifying RLVR environments.The method compares a student attempt with a correct solution on the same question.Source: §3 · p. 6, §3.2 · p. 7

3.1 Experimental setting

• The evaluation covers science Q&A and tool-use tasks with Qwen3-8B and Olmo3-7B-Instruct, reporting avg@16 relative to wall-clock training time, excluding initialization and validation.Science Q&A uses undergraduate-level reasoning subsets from SciKnowEval, while tool use maps tool specifications and user requests to correct calls using ToolAlpaca.Source: §3.1 · p. 6

3.2 Results

• SDPO outperforms GRPO across almost all runs, reaching 5-hour GRPO accuracy in 50 minutes on Olmo3-7B-Instruct and exceeding its 5-hour accuracy by more than 10 percentage points.The comparison uses avg@16 within one and five hours of wall-clock training time.Source: §3.2 · p. 7, Table 3 · p. 7
• The reported SDPO results use strictly on-policy training, leaving off-policy multi-update variants as future work.Each SDPO generation batch receives one gradient step, unlike off-policy methods with multiple updates per batch.Source: §3.2 · p. 7

3.3 Self-distillation learns to reason concisely

• SDPO consistently generates substantially shorter responses than GRPO while achieving higher accuracy, with responses more than 3× shorter on average.On Chemistry with Olmo3-7B-Instruct, response length is reduced 11× relative to GRPO while maintaining higher accuracy.Source: §3.3 · pp. 7–8
• SDPO’s shorter reasoning avoids filler phrases and circular repetition, which the paper attributes to token-level advantages from dense credit assignment.GRPO often repeats phrases such as “Hmm” and “Wait” or re-enters previous logical steps, whereas SDPO remains concise.Source: §3.3 · p. 8, Figure 7 · p. 8

4 Learning with Rich Environment Feedback

• 48.8% final accuracy lets SDPO outperform GRPO (41.2%), Claude Sonnet 4 (40.5%), and Claude Opus 4 (39.7%), while matching GRPO in 4× fewer generations.SDPO particularly improves over GRPO on medium and hard questions.Source: §4 · p. 9

4.1 Self-distillation benefits from stronger models

• SDPO’s gains increase with model strength: it significantly outperforms GRPO on larger Qwen3 models, but underperforms GRPO on Qwen2.5-1.5B.The results suggest accurate self-teacher retrospection emerges with scale.Source: §4.1 · p. 9, Figure 8 · p. 9

4.2 Self-distillation performs dense credit assignment

• Logit-level SDPO’s dense credit assignment significantly outperforms token- and sequence-level variants, while sequence-level SDPO still outperforms GRPO by leveraging rich feedback.SDPO assigns distinct advantages across possible next tokens, approximated using the top-K tokens plus the tail.Source: §4.2 · p. 10, Figure 10 · p. 11

4.3 The self-teacher improves during training

• The self-teacher improves significantly during training, enabling the student to surpass the initial teacher; regularized teachers outperform non-regularized and frozen alternatives.SDPO therefore bootstraps from a weak initial model without its performance limiting the final student.Source: §4.3 · pp. 10–11, Table 4 · p. 10, Figure 10 · p. 11

4.4 On-policy self-distillation avoids catastrophic forgetting

• SDPO learns LCBv6 while mitigating degradation on IFEval, ArenaHard-v2, and MMLU-Pro, achieving a better performance–forgetting tradeoff than GRPO.SFT on successful self-teacher generations significantly underperforms SDPO on LCBv6 and causes worse forgetting.Source: §4.4 · p. 11, Table 5 · p. 12

4.5 Can GRPO and SDPO be combined?

• SDPO+GRPO is more robust than SDPO on weaker models such as Qwen3-0.6B, but slightly underperforms SDPO on stronger models such as Qwen3-8B.The scalar-reward GRPO signal can stabilize unreliable SDPO advantages in weaker models but harm training with strong initial models.Source: §4.5 · p. 12, Figure 11 · p. 12

4.6 Which feedback is most informative?

• Environment outputs and student-generated sample solutions provide complementary informative feedback, whereas including the original attempt biases the teacher and reduces exploration.Environment output remains useful when the student has never solved the question, and reprompting-template syntax generally has little effect.Source: §4.6 · p. 13, Table 6 · p. 13

5.1 Experimental setting

• The experiment evaluates 19 hard and 9 very hard LCBv6 questions selected using Qwen3-8B pass@64 thresholds, retaining questions solved by best-of-k, multi-turn, or SDPO within 512 steps across five seeds.Hard tasks have pass@64 < 0.5, while very hard tasks have pass@64 < 0.03.Source: §5.1 · p. 14

5.2 Results

• SDPO substantially outperforms best-of-k and multi-turn sampling in discovery@k across hard and very hard LiveCodeBench questions.On very hard tasks, SDPO reaches 53.2% discovery@2750 versus 35.6% for multi-turn and 41.5% for best-of-k; on hard tasks, it reaches 78%.Source: §5.2 · p. 15, Figure 13 · p. 15
• SDPO reaches comparable discovery probabilities with substantially fewer generations, requiring approximately 3× fewer attempts than baselines on very hard tasks and roughly 2.4× fewer on hard tasks.SDPO reaches 22% discovery on very hard tasks with approximately 3× fewer generations, and 67% discovery on hard tasks with roughly 2.4× fewer generations.Source: §5.2 · p. 15
• SDPO uniquely solves a question that neither baseline solves within 2750 attempts, first discovering it after 321 attempts through 20 self-distillation iterations.SDPO also solves every question solved by best-of-k and multi-turn sampling.Source: §5.2 · p. 15
• Although the initial self-teacher solves fewer than 1% of almost all questions, its feedback-based credit assignment lets SDPO iteratively refine the policy and eventually solve them.The initial self-teacher has exactly 0% accuracy on 78% of questions, showing that one in-context feedback turn is insufficient.Source: §5.2 · pp. 15–16

5 Solving Hard Questions via Test-Time Self-Distillation

• Unlike binary-reward RLVR, which learns only after finding a solution, SDPO uses rich feedback after each attempt to correct mistakes before discovery.SDPO distills feedback-conditioned predictions into the policy, compressing interaction history into model weights rather than extending the context indefinitely.Source: §5 · p. 14, Figure 12 · p. 14, §5.2 · p. 16

6.1 Reinforcement Learning with LLMs

• RLVR methods such as STaR and GRPO use Monte Carlo scalar rewards, while value-network approaches add memory costs without removing the scalar-reward bottleneck.Large-scale RL has improved LLM reasoning, but scalar outcome rewards provide limited credit assignment.Source: §6.1 · p. 16

6.2 Learning from Rich Feedback and through Retrospection

• Rich-feedback methods either translate feedback into token- or state-level rewards with external models, or use feedback for in-context improvement, preference training, and answer-conditioned trace bootstrapping.Feedback-conditioned policies treat feedback as a deployment goal or relabeling signal, whereas RLRF uses feedback as a state for credit assignment in negative trajectories.Source: §6.2 · pp. 16–17

6.3 Distillation

• Distillation transfers teacher output distributions or representations, with recent on-policy methods using external-teacher feedback on student generations to reduce train-test mismatch.These approaches relate to online imitation learning but assume a distinct teacher model.Source: §6.3 · p. 17

6.4 Self-Distillation

• Self-distillation compresses extra context or environment feedback into model weights, while SDPO instead uses on-policy self-teaching to avoid mistakes in the model’s own generations.Off-policy self-distillation trains on teacher-generated samples and substantially underperforms SDPO; concurrent work studies scalar-reward grid worlds or fixed expert demonstrations.Source: §6.4 · p. 17

7 Conclusion, Limitations, and Future Work

• SDPO uses the current policy as a feedback-conditioned self-teacher, distilling corrected log-probabilities into the student for dense credit assignment.The method is a minimal, drop-in modification to standard RLVR pipelines.Source: §7 · p. 17
• SDPO achieves superior sample efficiency and wall-clock convergence over GRPO on reasoning tasks, including standard RLVR environments without rich feedback.Its gains increase with model scale, consistent with stronger in-context self-correction capabilities.Source: §7 · p. 17
• At test time, SDPO accelerates solution discovery on individual difficult binary-reward tasks compared with strong baselines.This extends SDPO beyond training-time optimization to per-question inference.Source: §7 · p. 17
• SDPO converts diverse tokenized feedback into effective self-supervision by retrospectively determining how the model should have acted.This approach uses precise outcomes rather than only binary rewards, resembling a more cognitively informed learning process.Source: §7 · pp. 17–18
• SDPO depends on in-context learning ability and feedback quality, may underperform GRPO on weaker models, and can fail with uninformative or misleading feedback.These dependencies make SDPO primarily applicable to reinforcement-learning stronger base models.Source: §7 · p. 18
• SDPO adds a small computational overhead for retrospective-model log-probabilities, which may matter more for smaller models with shorter generations.The overhead is often negligible but becomes relatively larger when generation time is comparatively small.Source: §7 · p. 18
• Future work should evaluate SDPO in long-horizon agentic settings, scale it to large multitask runs and frontier models, and test it beyond verifiable rewards.Open directions include open-ended text generation, continuous-reward tasks, and systematic analysis of reasoning behavior and reprompt templates.Source: §7 · p. 18

A Implementation of SDPO

• SDPO reprompts the current policy with feedback, distills its predictions through per-token divergences, and can use reverse-KL, forward-KL, or Jensen–Shannon objectives.The implementation also filters student and teacher distributions to top-K logprobs, includes a tail term, and can apply importance-sampling corrections for off-policy data.Source: Figure 14 · p. 29, A Implementation of SDPO · p. 29

A.1 Gradient Estimators

• The per-token gradient estimator sums tokenwise KL gradients while treating the sampling distribution that generated the rollout as fixed.A sequence-level estimator additionally accounts for how sampled tokens alter future prefixes, but experiments found no measurable gains sufficient to justify its extra complexity.Source: §A.1 · pp. 29–30

A.2 Regularized teacher

• SDPO stabilizes its changing self-teacher by constraining it toward the initial teacher with either an explicit trust region or an exponential moving average.The explicit teacher mixes reference and current log-probabilities using α, whereas EMA updates teacher parameters as θ′ ← (1 − α)θ′ + αθ.Source: §A.2 · p. 30
• The two teacher-regularization choices trade resources differently: EMA uses extra GPU memory without runtime overhead, while trust-region regularization adds log-probability computation.Trust-region regularization can avoid extra memory when the reference model is already used for explicit KL regularization.Source: §A.2 · p. 30

A.3 Approximate Logit Distillation

• Top-K logit distillation keeps only the student’s most likely tokens, avoiding nearly all memory overhead without significantly affecting performance.Without this approximation, both teacher and student logits must remain in memory; most vocabulary tokens are not informative at a given step.Source: §A.3 · pp. 30–31

A.4 Off-Policy Training: Generalization to Logit-Level Losses

• SDPO generalizes PPO-style off-policy corrections to logit-level losses by assigning each possible next token a per-logit advantage weighted by its old-policy probability.The formulation can sum over all tokens or the K most likely under πθold, replacing a Monte Carlo expectation estimate with explicit probability weighting.Source: §A.4 · p. 31
• In SDPO experiments, truncated importance sampling is applied at the token level rather than at the proposed logit level.The broader extension is described alongside PPO-style clipping, clip-higher, and fixed-length normalization methods.Source: §A.4 · p. 31

B Theoretical Analysis

• The theoretical analysis defines the self-teacher as qθ(· | x, f) := πθ(· | reprompt(x, f)) and derives SDPO’s gradient from Proposition 2.1.The proof uses the negative score function and the score trick to simplify the gradient.Source: B Theoretical Analysis · p. 32, §B.1 · p. 32

B.2 Trust-region Teacher

• The trust-region teacher constrains q to remain near the initial teacher qθref while optimizing proximity to the target qθ.Its closed-form solution combines log qθref and log qθ with weighting parameter α.Source: §B.2 · pp. 32–33

C Additional Related Work

• The related-work discussion contrasts SDPO with scalar-reward value methods, costly extra generations, external reward models, and RL pipelines that discard rich feedback.It frames verifiable LLM environments as partially observable because execution reveals latent states through feedback.Source: C Additional Related Work · p. 34

C.1 SDPO as Maximum Entropy RL

• SDPO is equivalent to maximum-entropy reinforcement learning with dense implicit reward r(yt | x, y<t) = log q(yt | x, f, y<t), linking self-distillation to inverse reinforcement learning.The retrospective model defines the implicit reward that the student learns.Source: §C.1 · pp. 34–35

D Additional Results & Ablations

• The additional-results appendix organizes experiments and ablations corresponding to the paper’s main learning settings, including results for Sections 3, 4, and 5.It also identifies comparisons of optimal hyperparameters and response lengths as appendix analyses.Source: D Additional Results & Ablations · p. 36, §D.1 · p. 36

D.1 Learning without rich environment feedback

• Table 7 compares SDPO and GRPO on reasoning benchmarks under independently selected model/task hyperparameters, while Table 8 reports their average response lengths.Table 7 reports highest avg@16 within one and five hours; both methods use one on-policy gradient step per generation batch, while GRPO also uses four off-policy steps.Source: Table 7 · p. 36, Table 8 · p. 36
• Figure 16 evaluates how train batch size and rollout count affect Qwen3-8B SDPO and GRPO accuracy on LCBv6 across three seeds.The figure reports pass@1 accuracy for batch sizes 4, 8, 16, and 32 and rollout counts 4 and 8, with standard errors.Source: Figure 16 · p. 37

D.2.1 Additional Results

• SDPO significantly improves over GRPO on medium and hard LCB questions, highlighting the value of rich feedback for challenging tasks.Figure 15 reports average accuracy during training through step 80, with standard deviations across three seeds.Source: §D.2.1 · p. 37, Figure 15 · p. 37
• Additional experiments vary GRPO and SDPO batch sizes and rollout counts, and evaluate Qwen2.5-Instruct alongside Qwen3 on LCBv6.The model-size comparison reports average validation accuracy with standard deviations across three seeds.Source: §D.2.1 · p. 37, Figure 17 · p. 38

D.2.3 Baselines

• Table 9 compares SDPO on LCBv6 with two GRPO variants, GSPO, and CISPO using Qwen3-8B through training step 80.Reported values include standard deviations across three seeds.Source: §D.2.3 · p. 37, Table 9 · p. 39

D.2.2 Training Stability

• Training diagnostics track SDPO’s loss, entropy, average gradient norm, and average response length on LCBv6.These metrics are logged throughout SDPO training.Source: Figure 18 · p. 38

D.3 Test-time self-distillation

• Up to 13.6× faster discovery than best-of-k on individual questions, SDPO reaches successes faster on average than best-of-k and multi-turn sampling.Means are truncated at 2,750 generations, with very hard tasks marked when pass@64 < 0.03.Source: Table 10 · p. 39
• Despite near-zero initial teacher accuracy on most hard and very hard tasks, self-distilled token-level advantages let SDPO iteratively refine its policy and solve them.Teacher accuracy measures successful single-turn solutions after reprompting the base model with feedback.Source: Table 11 · p. 40
• Including only prior feedback substantially outperforms concatenating full conversations in multi-turn sampling, while larger SDPO batch sizes yield more stable discovery-rate improvements as budgets grow.Smaller batches can discover slightly earlier below k < 26, but larger batches improve stability and scaling.Source: Figure 19 · p. 40
• On most of 19 hard questions, SDPO discovers solutions earlier than both the base model and multi-turn sampling, including Q3 after 321 attempts when both baselines remain at zero.Discovery@k curves show means and 90% confidence intervals across five random seeds per question.Source: Figure 20 · p. 41

E.1 Technical setup

• Experiments ran on one node with four NVIDIA GH200 GPUs totaling 378GB VRAM, using CUDA 12.8 and PyTorch v2.7.0.The environment used the NVIDIA PyTorch 25.02 container.Source: §E.1 · p. 42
• The implementation used verl with PyTorch FSDP2 for distributed training and vLLM for efficient batched rollout generation.These components supported training and inference on the multi-GPU node.Source: §E.1 · p. 42

E.2 Hyperparameters

• SDPO and GRPO hyperparameters were summarized separately in Tables 12 and 13 for each experimental setup.The tables provide the method-specific configurations used in the experiments.Source: §E.2 · p. 42, Table 12 · p. 42, Table 13 · p. 43

E.2.1 Details on Hyperparameter Selection (Section 3)

• For Section 3, each method’s single configuration was selected by validation accuracy within five hours across all datasets and models.The search varied learning rates, minibatch sizes, and, for SDPO, forward-KL versus Jensen–Shannon variants; per-dataset/model optima were also reported in Table 3.Source: §E.2.1 · p. 43

E.3 User Templates

• Multiple-choice and tool-use experiments used task-specific prompt templates, including structured tool descriptions, JSON outputs, and explicit Thought–Action–Action Input formatting.The tool-use example invokes an Axolotl API through a user request for a random picture.Source: §E.3 · p. 44

F.1 Visualization of Advantages

• Figure 21 compares SDPO and GRPO token-level advantages in an Olmo3-7B-Instruct Chemistry batch, visualizing positive and negative values by color.Blue denotes positive advantages and red denotes negative advantages at response positions.Source: §F.1 · p. 46, Figure 21 · p. 46

F.2 Examples

• The qualitative examples include SDPO training on LCBv6 with Qwen3-8B, a binary-string programming task, and feedback cases labeled Wrong Answer, Memory Error, and Index Error.The coding example shows prompt, problem, candidate reasoning, and implementation details for maximizing active sections after one trade.Source: §F.2 · pp. 46–48, §F.3 · pp. 49–50

F.4 Illustrative Example

• Figure 22 shows SDPO converting feedback into sparse, per-token credit assignment, while GRPO would assign the same negative advantage to every token under binary rewards.The self-teacher retrospectively identifies an error and alternative solution path without an explicit solution, reinforcing or revising only specific token positions.Source: §F.4 · p. 50, Figure 22 · p. 50