PaperlayerView exact revision on arXiv

Source-linked AI summary

Does self-distillation improve reasoning?

Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma, Avinash Balakrishnan, Rameswar Panda, Polina Golland, Yoon Kim, Zhang-Wei Hong

arXiv:2610.18613v1cs.LGcs.AIcs.CL

TL;DR

Verifiable reasoning rewards often give only a binary outcome, while self-distillation is intended to use richer privileged information; whether its gains generalize and come from that information remains unclear. This paper evaluates SDPO and OPSD across reasoning tasks and finds inconsistent transfer and weak alignment with policy gradients. Its results leave effective use of privileged information for reasoning as an open challenge.

  • Problem

    It remains unclear whether self-distillation gains extend beyond in-distribution evaluations and stem from privileged information rather than other mechanisms.

    Source: §1 · pp. 1–2

  • Method

    The paper revisits SDPO and OPSD across coding, science, tool-use, math, and gradient-alignment evaluations.

    Source: §1 · p. 2

  • Results

    Self-distillation gains do not consistently generalize or derive from privileged information, and its gradients are less aligned with held-out policy gradients than GRPO’s.

    Source: §1 · p. 2, §6 · pp. 9–10

  • Takeaways & Limitations

    Effectively leveraging privileged information to improve reasoning remains an open challenge.

    Source: §6 · p. 10

    Experiments cover only Qwen3 and Olmo model families, SDPO and OPSD recipes, and verifiable domains, not newer self-distillation techniques or non-verifiable settings.

    Source: §5 · p. 9

Abstract

from arXiv · show

Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as correct answers or environmental feedback. Recent methods have reported better sample efficiency and generalization than reinforcement learning (RL), but it remains unclear whether these gains extend beyond in-distribution evaluations and whether they stem from privileged distillation. In this paper, we revisit two representative methods, Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD), across code, science, tool use, and math reasoning tasks. We find that their reported gains do not consistently generalize to unseen problems and are largely explained by overfitting or transferring existing reasoning behavior, rather than effectively using privileged information. Moreover, self-distillation gradients show weak alignment with policy gradients that directly optimize expected reward, questioning whether they provide an effective direction for policy improvement. Our experiments across Qwen3 models (1.7B-14B) and OLMo-3-7B suggest that effectively leveraging privileged information to improve reasoning remains an open challenge. Our code is available at https://github.com/sleepypandu/does-sd-improve-reasoning

1 INTRODUCTION

Verifiable rewards often report only whether a final answer is correct, while self-distillation uses richer feedback to provide token-level training signals. The paper revisits whether that privileged information explains reported gains from self-distillation.

1 INTRODUCTION

  • SDPO’s coding gains do not transfer to unseen problems: it overfits training problems, reduces solution diversity, and degrades on held-out problems, unlike policy-gradient training.The diversity comparison counts AST-unique solutions per training problem from eight samples.Source: Figure 1 · p. 1, §1 · p. 2
  • Across five science and tool-use domains, SDPO and GRPO perform similarly in-domain, but SDPO lowers held-out accuracy in four while GRPO preserves it.Source: §1 · p. 2
  • In math, privileged information barely changes OPSD’s teacher distribution, while transferring thinking-mode behavior has a substantially larger effect.Source: §1 · p. 2, Figure 2 · p. 2
  • On Qwen3 models, SDPO and OPSD gradients are less aligned with held-out policy gradients than GRPO gradients, and often point away from them.The comparison uses GRPO gradients on the same training problems as the reference.Source: §1 · p. 2

2 RELATED WORKS

Related work contrasts reward-based optimization with distillation approaches and describes concerns about whether self-distillation reliably improves reasoning.

2 RELATED WORKS

  • RLVR uses verifiable trajectory-level rewards for tasks such as math and coding, while process reward models score individual reasoning steps but require additional models.Source: §2 · p. 3
  • On-policy distillation minimizes per-token divergences along student-sampled prefixes and has been used for small-model post-training and merging specialist capabilities.Source: §2 · p. 3
  • Self-distillation conditions a copy of the student on privileged information, such as a reference solution or environmental feedback, to purportedly transfer reasoning content.Source: §2 · p. 3
  • Prior studies report poor out-of-distribution generalization, suppressed backtracking, collapsed response lengths, and difficulty distinguishing correct from incorrect generations.Some reported OPSD gains instead stem from thinking-mode transfer; advantage-reweighting methods fall outside this paper’s objective-focused scope.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define supervised fine-tuning, policy-gradient training, and privileged-information self-distillation. They then show that these update directions coincide under an optimal-teacher assumption, motivating an empirical alignment test.

3 PRELIMINARIES

  • GRPO samples student rollouts, scores them with a reward function, and uses sequence-level normalized advantages in a policy-gradient update.The update aims to maximize expected reward.Source: §3 · pp. 3–4
  • Privileged-information self-distillation matches student predictions along sampled traces to a teacher conditioned on feedback, with SDPO and OPSD differing in divergence, clipping, and feedback source.SDPO uses reverse KL over top-k tokens with sibling rollouts and environment feedback; OPSD uses clipped forward KL and reference solutions.Source: §3 · p. 4

3.1 EQUIVALENCE BETWEEN POLICY GRADIENT, SFT, AND DISTILLATION

  • For the KL-regularized optimal teacher, policy gradients, supervised fine-tuning on its samples, and on-policy distillation share the same update direction.The teacher maximizes expected reward minus a KL penalty relative to the current policy.Source: §3.1 · p. 4
  • The equivalence motivates testing gradient alignment because self-distillation’s benefit depends on its teacher, which must remain close to the student.A teacher far from the student forfeits the small-KL-shift property discussed for on-policy RL.Source: §3.1 · p. 4

4 EXPERIMENTAL RESULTS

Across coding, science and tool-use, math, and gradient-alignment experiments, the authors test whether self-distillation generalizes and uses privileged information. The results show limited transfer to unseen problems and indicate that OPSD gains can instead reflect thinking-mode transfer.

4.1 SDPO DOES NOT GENERALIZE TO NEW CODING PROBLEMS

  • On TEST UNSEEN, GRPO gains 3.7, 4.9, and 2.6 points, while SDPO changes −2.5, −0.5, and −9.9 on Qwen3-8B, Qwen3-14B, and Olmo-3-7B-Instruct, respectively.SDPO ends 13.1 points below GRPO on Olmo-3-7B-Instruct (21.0 vs. 34.1).Source: Figure 3 · p. 5, §4.1 · p. 5
  • On training problems, SDPO’s gains on seen test cases largely carry over to held-out test cases, but it produces fewer distinct solutions and fails to improve on new problems.With longer training, GRPO overtakes SDPO on Qwen3 training-problem evaluations after about 160 steps; SDPO continues to outperform GRPO on Olmo-3-7B-Instruct training problems.Source: §4.1 · pp. 5–6

4.2 SDPO DOES NOT GENERALIZE OOD IN SCIENCE REASONING AND TOOL USE

  • Training SDPO on science domains lowers average out-of-domain accuracy by 7.1–16.6 points, while GRPO improves it by 0.9–1.6; tool-use training is an exception.Across the five domains, in-domain accuracy is similar: 66.1 for SDPO and 67.0 for GRPO.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 OPSD GAINS DEPEND ON THINKING MODE TRANSFER, NOT PRIVILEGED INFORMATION

  • The correct reference solution was best in only 3 of 18 OPSD configurations, each by under one point; variants without it or with shuffled solutions often matched it.On a sample rollout, removing the reference solution barely changes the self-conditioned teacher’s token probabilities.Source: §4.3 · pp. 7–8
  • Thinking-on teachers, rather than reference-solution content, drive OPSD’s thinking-off gains and induce thinking-style outputs and longer responses.Across Qwen3-1.7B to Qwen3-14B, enabling teacher thinking raises average thinking-off accuracy by 7.6, 22.9, 17.3, and 28.9 points.Source: §4.3 · p. 8

4.4 SELF-DISTILLATION GRADIENTS DO NOT ALIGN WITH POLICY GRADIENTS

  • All 24 OPSD variants have negative cosine similarity with the held-out math policy gradient, whereas GRPOnothink is positive across Qwen3-1.7B to 14B.For Qwen3-8B, the table reports cosine similarity with the held-out GRPOnothink gradient.Source: §4.4 · p. 9, Table 3 · p. 9
  • On all three coding models, SDPO gradients align less with the held-out policy gradient than GRPO; self-distillation gradients also have lower variance across seeds.SDPO points away from the reference on Qwen3-8B; its and OPSD’s gradient noise scale is at most 0.05, versus 1.97 to 4.11 for GRPO.Source: Table 4 · p. 9, §4.4 · p. 9

5 LIMITATIONS

The experiments cover only Qwen3 and Olmo models, SDPO and OPSD recipes, and verifiable domains; newer self-distillation methods and non-verifiable use cases remain outside scope.

5 LIMITATIONS

  • Experiments are limited to the Qwen3 and Olmo model families and the SDPO and OPSD recipes.The paper does not address newer harness, contrastive, or neighboring-rollout feedback approaches, which may combat these methods’ limitations.Source: §5 · p. 9
  • The study focuses on verifiable domains, although self-distillation also has use cases in non-verifiable settings.Source: §5 · p. 9

6 CONCLUSION

The paper concludes that privileged self-distillation gains either fail to generalize or do not derive from privileged information, and that its gradients do not align with policy gradients on held-out problems. It situates these findings alongside related evaluations and diagnostics while calling for principled reinforcement learning methods that exploit privileged information.

6 CONCLUSION

  • SDPO’s coding gains are confined to training problems, its other-domain gains come at an out-of-distribution cost, and OPSD’s math gains rely on thinking-mode transfer rather than privileged information.Self-distillation gradients also fail to align with the policy gradient on held-out problems.Source: §6 · pp. 9–10
  • The authors call for principled reinforcement learning methods that can exploit privileged information.Source: §6 · p. 10
  • The authors report using generative AI tools across research, analysis, writing, and coding tasks, with authors reviewing AI-assisted work and checking retrieved examples.Source: AI USE STATEMENT · p. 10
  • Code is publicly available, and appendices detail experimental setups, hyperparameters, OPSD prompts, and the gradient-similarity method.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXTENDED RELATED WORKS

  • Prior empirical re-evaluations report SDPO generalization and continual-learning problems, while concurrent work finds OPSD gains rely on thinking-mode transfer; this paper emphasizes exact reproductions and SDPO reasoning gains.Source: A EXTENDED RELATED WORKS · p. 15
  • Prior token-level studies link self-distillation losses on uncertain, high-entropy tokens to suppressed backtracking and shorter responses; this paper instead diagnoses OPSD improvements as thinking-mode transfer.Source: A EXTENDED RELATED WORKS · p. 15
  • Earlier sequence-level work finds self-conditioned teachers struggle to distinguish correct from incorrect solutions or favor reference solutions, while this paper tests whether they approximate the optimal KL-regularized model.Source: A EXTENDED RELATED WORKS · p. 15
  • Methods that retain GRPO and use the self-conditioned teacher to modulate token advantages fall outside this study’s self-distillation-loss-only scope.Their use of GRPO is consistent with work questioning whether the self-distillation objective alone yields performance gains.Source: A EXTENDED RELATED WORKS · p. 15

B PROOF OF PROPOSITION 3.1

This appendix restates Proposition 3.1’s gradient result and provides additional context on Qwen3 thinking modes and SDPO evaluation. Its figures and tables describe training and cross-domain evaluations, without giving their score values in these passages.

B PROOF OF PROPOSITION 3.1

  • For binary rewards and β > 0, the negated gradients of both KL objectives are positive multiples of the reward gradient when that gradient is nonzero.Normalizing them to unit ℓ2 norm yields Equation (4).Source: B PROOF OF PROPOSITION 3.1 · pp. 15–16

C QWEN3 THINKING MODES

  • Qwen3 thinking mode reasons inside a <think>...</think> block, while non-thinking mode is trained to respond directly after an empty think block.Source: C QWEN3 THINKING MODES · p. 16

D.1 SDPO LIVECODEBENCH V6 PERFORMANCE

  • Additional LiveCodeBench v6 results report scores for Qwen3-8B, Qwen3-14B, and Olmo-3-7B-Instruct on TEST UNSEEN, TRAIN SEEN, and TRAIN UNSEEN at steps 0, 80, and 200.Figures 4 and 5 show the Qwen3-14B and Olmo-3-7B-Instruct training curves; the supplied passages do not give the score values.Source: §D.1 · p. 16, Table 5 · p. 17, Table 6 · p. 17

D.2 SDPO SCIENCE REASONING AND TOOL USE PERFORMANCE

  • Figure 6 plots cross-domain validation accuracy for Qwen3-8B trained with GRPO and SDPO on SciKnowEval and ToolAlpaca, with training domains as rows and evaluation domains as columns.It complements Table 1’s step-200 results, whose values are not included here.Source: §D.2 · p. 16, Figure 6 · p. 19

D.3 OPSD METRICS ON QWEN3 MODELS

The OPSD appendix supplies table descriptions for math accuracy and rollout statistics, alongside example generations. The examples contrast thinking-mode rollouts and show an unmatched closing think token in one thinking-off response.

D.3 OPSD METRICS ON QWEN3 MODELS

  • The OPSD tables report best-performance checkpoints for Qwen3-1.7B, Qwen3-4B, and Qwen3-14B under avg@12 thinking-on/off evaluation, and Olmo-3-7B-Instruct under avg@12; results average four seeds and report mean±std accuracy.Source: Table 8 · p. 20, Table 9 · p. 20, Table 10 · p. 21
  • Across Qwen3 models and Olmo-3-7B-Instruct, the rollout tables track response length and orphan </think> tokens; the Qwen3 tables pool AIME 24, AIME 25, and HMMT 25 evaluations.Tok@best and Tok@200 denote average response length at the best checkpoint and checkpoint 200, respectively.Source: Table 12 · p. 22, Table 13 · p. 22, Table 14 · p. 23

D.4 OPSD EXAMPLE GENERATIONS

  • The base Qwen3-8B thinking-off example is 559 tokens and uses one matched <think>...</think> pair, while the thinking-on example is 5,067 tokens and also uses one matched pair.Source: Figure 7 · p. 25, Figure 8 · p. 26
  • A 2,529-token Qwen3-8B thinking-off rollout trained with a thinking-mode teacher opens with an extended reasoning trace and emits an unmatched </think> before its polished solution.Source: Figure 9

D.5 OPSD MASKING RESULTS

The Qwen3-8B OPSD masking ablation varies which student-token blocks receive distillation loss and which privileged information the teacher sees. The supplied table caption describes the evaluation but does not provide outcome values.

D.5 OPSD MASKING RESULTS

  • A separate Qwen3-8B ablation confines losses to either the first 90% or the last 10% of the student rollout.Source: §D.5 · p. 28
  • The Qwen3-8B OPSD masking ablation evaluates checkpoints with avg@12 on AIME24, AIME25, and HMMT25, aggregating accuracy across four seeds.The caption does not include the table’s accuracy values.Source: Table 17 · p. 28
  • The ablation compares distillation loss on different blocks of student tokens, with the remaining tokens masked out.Source: Table 17 · p. 28
  • Within each block, the teacher receives the reference solution, a shuffled solution from another problem, or no privileged information.Teacher thinking was on throughout OPSD training.Source: Table 17 · p. 28

D.6 GRADIENT ALIGNMENT RESULTS

The paper reports gradient cosine similarities between self-distillation methods and GRPO gradients on math probes, alongside gradient-noise measurements. The supplied passages specify the comparisons but not their numerical outcomes.

D.6 GRADIENT ALIGNMENT RESULTS

  • Table 18 compares method gradients on OpenThoughts with GRPOnothink gradients on AIME24, AIME25, and HMMT25 for Qwen3 models.Cosine similarities are pooled over four seeds and reported with paired jackknife error.Source: Table 18 · p. 28, §D.6 · p. 28
  • Table 19 reports the same math-probe cosine comparison for Olmo-3-7B-Instruct, using GRPO gradients as the reference.OPSD self-distill uses the frozen instruction model as teacher; OPSD cross-model uses Olmo-3-7B-Think.Source: §D.6 · p. 28, Table 19 · p. 29
  • Tables 20 and 21 report gradient noise scales for the math and LiveCodeBench v6 probes, respectively.The math reference is GRPOnothink on AIME24, AIME25, and HMMT25; the coding reference is GRPO on TEST UNSEEN, while GRPO and SDPO train on TRAIN SEEN.Source: §D.6 · p. 28, Table 20 · p. 29, Table 21 · p. 29

E IMPLEMENTATION DETAILS

The implementation follows established configurations for SDPO and OPSD with specified dataset-specific changes. Experiments use single-node 8×H100 setups and the reported inference and training software.

E IMPLEMENTATION DETAILS

  • Experiments ran on 8×H100 nodes without multi-node parallelism, primarily using vLLM, with OPSD in TRL and SDPO in veRL.Reported runtimes vary by experiment: SDPO LCBv6 runs took about 40 hours, while OPSD runs took up to about two hours.Source: E IMPLEMENTATION DETAILS · p. 30

E.1 SDPO HYPERPARAMETERS

  • LiveCodeBench v6 SDPO uses the original rich-feedback settings and prompts, changing only the dataset splits, with hyperparameters summarized in Table 22.Source: §E.1 · p. 30, Table 22 · p. 30
  • For SciKnowEval and ToolAlpaca, SDPO and GRPO follow prior hyperparameters, but GRPO uses a 1 × 10−6 learning rate because 1 × 10−5 was unstable.Both methods run on-policy; hyperparameters are summarized in Table 23.Source: §E.1 · p. 30, Table 23 · p. 31

E.2 OPSD HYPERPARAMETERS

  • OPSD and SFT use OpenThoughts configurations from Zhao et al. (2026a), changing only z, and train 100 steps longer for fully converged evaluation.Table 24 summarizes the configurations.Source: §E.2 · p. 31

E.3 GRADIENT SIMILARITY ANALYSIS

The gradient analysis records fixed-weight training gradients and compares pooled method gradients with reference gradients using cosine similarity and noise measures. The OPSD prompts distinguish a student prompt from a teacher prompt that includes a reference solution.

E.3 GRADIENT SIMILARITY ANALYSIS

  • Each method’s gradients are averaged over training iterations from runs at learning rate zero, so the model weights remain unchanged.The analysis uses four seeds, with 64 iterations per seed for OPSD and eight for SDPO.Source: §E.3 · p. 31
  • Pooled cosine similarity compares the averaged gradients of the reference and each method across four seeds.The analysis defines jackknife error by recomputing pooled cosine similarity with each seed removed in turn.Source: §E.3 · pp. 31–32
  • Gradient noise scale measures noise in each pooled gradient, with zero representing a noiseless estimate.The estimate uses the covariance of the seed-level gradients.Source: §E.3 · p. 32

E.4 OPSD TEACHER AND STUDENT PROMPTS

  • The OPSD teacher prompt places a reference solution in the context, then asks the model to reason independently rather than copy or paraphrase it.The student prompt asks for step-by-step reasoning and a boxed final answer.Source: §E.4 · p. 33, Figure 11, Figure 10

Related papers

Loading 2610.18613v1…