PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, Yuqing Yang

arXiv:2603.24472v4cs.CLcs.LG

TL;DR

The paper asks why self-distillation can shorten reasoning while degrading mathematical performance, focusing on whether it suppresses uncertainty expression. Through controlled context-richness and training interventions, it finds that suppressed epistemic verbalization can improve compact in-domain optimization yet harm broader or OOD reasoning.

  • Problem

    Existing self-distillation results suggest shorter responses can improve performance, but mathematical reasoning can show the opposite pattern, raising questions about context richness, uncertainty expression, and task coverage.

  • Method

    The paper empirically varies conditioning-context richness, measures epistemic verbalization, and explicitly penalizes epistemic tokens during GRPO to examine their relationship with reasoning performance.

  • Results

    Self-distillation produces shorter, more confident reasoning and can improve limited-coverage in-domain performance, but becomes less effective with broad coverage and may harm OOD performance.

  • Takeaways & Limitations

    Robust post-training should preserve appropriate uncertainty-aware reasoning behavior rather than optimize only for answer correctness or shorter traces.

  • Takeaways & Limitations

    The experiments use publicly available models and datasets and do not involve new data collection or human subjects research.

Abstract

from arXiv · show

Self-distillation has emerged as an effective post-training paradigm for LLMs, often improving performance while shortening reasoning traces. However, in mathematical reasoning, we find that it can reduce response length while degrading performance. We trace this degradation to the suppression of epistemic verbalization - the model's expression of uncertainty during reasoning. Through controlled experiments varying conditioning context richness and task coverage, we show that conditioning the teacher on rich information suppresses uncertainty expression, enabling rapid in-domain optimization with limited task coverage but harming OOD performance, where unseen problems benefit from expressing uncertainty and adjusting accordingly. Across Qwen3-1.7B/8B, DeepSeek-Distill-Qwen-7B, and Olmo3-7B-Instruct, we observe performance drops of up to 40%. Our findings highlight that exposing appropriate levels of uncertainty is crucial for robust reasoning and underscore the importance of optimizing reasoning behavior beyond merely reinforcing correct answer traces.

1 Introduction

Self-distillation can shorten reasoning while improving performance in some domains but degrading mathematical reasoning. The analysis attributes this contrast to suppressed uncertainty expression and shows that information richness and task coverage shape the outcome.

  • Self-distillation reduces response length while improving Chemistry performance but reduces mathematical performance despite similarly shortening responses.The paper contrasts SDPO with GRPO across Chemistry and DAPO-Math-17k.
  • Richer teacher context produces more concise and confident reasoning with fewer expressions of epistemic uncertainty.The paper identifies suppression of epistemic verbalization as the mechanism behind this shift.
  • Limited task coverage enables rapid in-domain optimization, whereas broader coverage makes suppressed uncertainty harmful on challenging or unseen problems.The reported effect depends on both privileged-information richness and task coverage.
  • Standard objectives can preserve correct reasoning traces while allowing reasoning-style changes that hurt out-of-distribution generalization.The authors argue that post-training should consider uncertainty-aware reasoning behavior alongside answer correctness.

2 Preliminaries

Self-distillation uses one model as student and teacher under different conditioning contexts, transferring information from richer teacher inputs. In mathematical reasoning, verbalized uncertainty can preserve alternatives and support error recovery.

  • Self-distillation uses the same model as student and teacher under different conditioning contexts.The student generates from the original input, while the teacher receives additional information such as solutions or feedback.
  • Training minimizes divergence between student and teacher next-token distributions, transferring privileged information without an external teacher.The student is optimized to match predictions produced under the richer context.
  • Math reasoning iteratively updates beliefs over intermediate hypotheses across diverse task types, making uncertainty expression potentially useful.The paper frames this process as self-Bayesian reasoning conditioned on the prompt and prior tokens.
  • Epistemic verbalization preserves alternative hypotheses and supports gradual uncertainty reduction, whereas omitting it can cause premature commitment to incorrect hypotheses.The paper treats uncertainty markers as informative reasoning signals rather than mere stylistic redundancy.

3 LLM Reasoning Behavior Under Richer Information

The paper measures conditioning-context richness and tests how it changes reasoning behavior. Richer information shortens responses and suppresses epistemic markers, motivating controlled comparisons of guided and unguided generation.

  • Conditional mutual information I(y*; c | x) measures how much additional context reduces uncertainty about an ideal correct response.The measure is defined as H(y*|x) − H(y*|x,c).
  • The controlled experiment compares unguided, solution-guided, think-removed, and regeneration-conditioned responses on 100 DAPO-Math-17k problems.The base model is DeepSeek-R1-Distill-Qwen-7B, and selected problems have base accuracy between 0.125 and 0.5 over eight rollouts.
  • The four settings are ordered by conditioning information: unguided is lowest, solution guidance is highest, and think-removed or regenerated responses fall below full-solution guidance.The ordering follows subset and data-processing arguments described in the experiment design.
  • The study counts ten epistemic markers—wait, hmm, perhaps, maybe, actually, alternatively, seems, might, likely, and check—as practical uncertainty indicators.These markers operationalize epistemic verbalization for response-level analysis.
  • As conditioning information increases, response length and epistemic-token count decrease monotonically, producing more concise and confident reasoning.Unguided generation has the longest responses and highest epistemic-token counts; removing think content increases both toward the unguided level.

4 Supervised Finetuning with Self-Distillation

The paper tests whether the epistemic density of correct self-distillation targets affects mathematical reasoning. Solution-guided targets substantially degrade benchmark performance, while unguided targets produce no significant change.

  • The study performs off-policy self-distillation using 800 correct responses from each of two datasets: unguided and solution-guided.The checkpoints are evaluated across multiple mathematical benchmarks.
  • Unguided responses average about 12k tokens with high epistemic-token counts, whereas solution-guided responses average about 2k tokens with low counts.Both datasets contain correct trajectories; they differ in the epistemic density of their training signals.
  • Solution-guided training substantially degrades performance across all evaluated benchmarks despite using correct answers, while unguided training produces no significant performance change.The reported asymmetry links the training targets’ reasoning style to downstream mathematical performance.
  • Solution-guided targets force imitation of reasoning that presupposes unavailable information, suppressing epistemic tokens used for autonomous exploration and error correction.This provides the paper’s proposed explanation for the degradation.

5 On-Policy Self-Distillation

On-policy self-distillation can shorten reasoning while suppressing epistemic verbalization, with the effect depending on teacher context and the base model’s initial uncertainty expression. Across models and settings, stronger suppression is associated with larger performance degradation, especially OOD.

  • Experimental setup: On-policy experiments compare GRPO and SDPO across DeepSeek-R1-Distill-Qwen-7B and Qwen3-8B under different conditioning contexts and thinking modes.The study varies c = s versus c = s\think for DeepSeek-R1-Distill-Qwen-7B and examines Qwen3-8B with thinking mode on or off.
  • DeepSeek-R1-Distill-Qwen-7B: For DeepSeek-R1-Distill-Qwen-7B, SDPO with c = s sharply reduces response length and training score, while c = s\think attenuates the length drop and approaches GRPO’s score trajectory.The richer full-solution context produces the strongest degradation during training.
  • DeepSeek-R1-Distill-Qwen-7B: SDPO with c = s degrades OOD performance by ∼40% on AIME24 and ∼15% on AMC23, whereas GRPO improves AIME24 from 54.7 to 56.0 and AMC23 from 89.3 to 91.1.Reducing conditioning to c = s\think mitigates SDPO’s drop, but performance remains below the base model.
  • Epistemic verbalization: GRPO increases epistemic-token usage, whereas SDPO suppresses it more aggressively, matching the observed relationship between epistemic suppression and performance degradation.The reasoning-pattern comparison links changes in E[E(y)] to the differing outcomes.
  • Qwen3-8B: Thinking Mode ON: For Qwen3-8B in thinking mode, both methods reduce response length and epistemic verbalization, but SDPO’s larger reduction produces a larger performance drop and worse OOD results.On AIME24, SDPO progressively degrades despite comparable training performance under c = s\think.
  • Qwen3-8B: Thinking Mode OFF: Without thinking mode, GRPO rapidly lengthens responses through epistemic verbalization and gains training performance, while SDPO shortens responses, improves slowly, and slightly reduces AIME24 performance from 0.25 to 0.23.This setting further illustrates the cost of suppressing uncertainty expression.
  • Causal study: Explicitly penalizing epistemic tokens during GRPO produces shorter responses and lower performance, qualitatively reproducing SDPO’s degradation pattern.The intervention provides causal evidence that suppressing uncertainty expression contributes to SDPO’s performance decline.
  • Moving-target teacher: Even slow EMA teacher updates amplify epistemic suppression through a feedback loop and cause greater degradation than a fixed teacher.A fixed teacher uses EMA rate 0.0, which yields better performance.

6 Relationship Between Task Coverage, Epistemic Verbalization and Generalization Ability

Task coverage determines whether concise, confident self-distilled reasoning is beneficial or harmful. Limited coverage enables rapid in-domain optimization, whereas broader and more diverse problem sets make epistemic verbalization increasingly important for generalization.

  • Motivation: Prior settings report that SDPO improves accuracy while shortening responses, but the math-focused setup shows that this pattern depends on task coverage.Science Q&A and LiveCodeBench provide contrasting coverage conditions relative to DAPO-Math-17k.
  • Experimental design: The study varies the training-set size |D| over {1, 8, 64, 128, 512} using DAPO-Math-17k and compares GRPO with SDPO on Qwen3-8B in thinking mode OFF.This isolates the relationship between task coverage and generalization.
  • Training dynamics: When |D| ≤128, SDPO quickly reaches high scores while reducing E[L(y)] by up to 8×, indicating greater training efficiency on small task sets.At |D| = 512, further response-length reductions begin to hurt SDPO’s training score relative to GRPO.
  • Training dynamics: As task coverage grows, GRPO increases E[E(y)] to express uncertainty and adapt reasoning, while SDPO’s confident concision becomes limiting for larger, more diverse problem sets.The methods therefore use different reasoning responses to increasing task diversity.
  • OOD evaluation: On AIME24 and MATH500, GRPO performance scales with |D|, whereas SDPO suffers more severe OOD degradation at smaller |D| and remains below the base model even at the largest |D|.Larger training sets progressively improve GRPO alongside increasing E[L(y)].
  • Takeaway: Epistemic verbalization is largely redundant for familiar repetitive tasks but increasingly important as task diversity and generalization demands grow.The paper frames uncertainty expression as removable for efficiency only when coverage is limited.

7 Conclusion

Self-distillation produces shorter, more confident reasoning and can quickly improve limited-coverage in-domain performance, but becomes less effective with broad coverage and may harm OOD performance. The findings show that preserving reasoning behavior beyond answer correctness matters for robust optimization.

  • Self-distillation tends to produce higher-confidence answers and shorter reasoning traces.
  • Limited task coverage can make compressed reasoning rapidly improve in-domain performance.
  • Broad task coverage reduces self-distillation’s effectiveness and may harm out-of-distribution performance.
  • Correct-reasoning objectives alone may not preserve robust reasoning because optimization can reshape reasoning behavior beyond answer correctness.

Ethics Statement

The study uses publicly available models and datasets without new data collection or human subjects research. It identifies suppressed uncertainty expression as a concern because incorrect answers may be produced with unwarranted confidence.

  • The experiments use publicly available Qwen3, DeepSeek-R1-Distill-Qwen, and OLMo models with DAPO-Math-17k, AIME, AMC, and MATH500 datasets.
  • The study involves no new data collection or human subjects research, so privacy and human-subjects concerns do not apply.
  • Self-distillation can suppress epistemic verbalization and diminish uncertainty expression during reasoning.
  • Indiscriminately suppressing uncertainty expressions may increase the risk of incorrect answers delivered with unwarranted confidence.

Reproducibility Statement

The paper situates its analysis within self-distillation, uncertainty expression, and reasoning-compression research, arguing that compression can harm OOD generalization when it removes uncertainty signals. It emphasizes retaining epistemic expressions while filtering non-informative content.

  • Related work: Self-distillation uses asymmetric conditioning so one model instance provides dense reward signals to another instance lacking privileged information.
  • Related work: Prior self-distillation work demonstrates effectiveness but does not explain when or why performance degrades on problems solved without external-environment interaction.
  • Uncertainty Expression in LLM Reasoning: The paper identifies conditioning-context richness and task coverage as factors governing whether self-distillation suppresses epistemic verbalization.
  • Reasoning Compression: Reasoning compression that suppresses uncertainty expression can harm OOD generalization, with the effect modulated by task coverage.
  • Reasoning Compression: The proposed challenge is to remove non-informative content while retaining epistemic expressions that support iterative belief refinement.

B Basis for Selecting Epistemic Tokens

The paper establishes epistemic-token indicators through cross-model trace analysis, then documents evaluation and training settings and several behavioral comparisons. These materials support measuring uncertainty expression and examining how distillation choices affect reasoning performance.

  • Token selection: The study analyzes 9,600 reasoning traces from eight models spanning Qwen3, QwQ, DeepSeek-R1-Distill-Qwen, and OLMo families.
  • Token selection: The traces come from 600 math problems drawn from AIME 2024/2025, AMC 2023, and MATH500, with two samples per problem for each model.
  • Token selection: An LLM-based extraction prompt identifies uncertainty spans and their one-to-three-word trigger tokens.
  • Token selection: The selected markers cover 75–93% of occurrences of the corresponding uncertainty phrases in Table 4.
  • Evaluation and ablations: Under a 4,096-token constraint, SDPO gains training scores earlier, but GRPO eventually achieves higher AIME24 and AMC23 performance.
  • Evaluation and ablations: Reducing the learning rate from 1e−5 to 1e−6 slows degradation but ultimately leads to the same degraded reasoning behavior.
  • Epistemic-token behavior: SDPO reduces epistemic-token usage relative to the base model, especially when task coverage |D| is smaller.

E.4 Comparative Analysis of Epistemic and Overall Word Usage

The analysis finds that training selectively changes epistemic expressions rather than uniformly shifting vocabulary usage. SDPO also suppresses uncertainty expressions and diverges most from the student at tokens initiating epistemic deliberation.

  • Word-usage comparison: Epistemic-token frequency shifts are disproportionately larger than shifts across the overall vocabulary.The comparison tests whether training specifically changes uncertainty-related expressions rather than inducing general vocabulary drift.
  • Word-usage comparison: Average per-word shifts across the full vocabulary remain near zero, while epistemic-token shifts reach 30×–40× larger magnitudes, including −11.9 and −12.2 for SDPO.The reported full-vocabulary average satisfies |∆| < 1.
  • Uncertainty-expression measurement: LLM-as-Judge annotation captures multi-token uncertainty phrases and supports epistemic markers as a practical proxy for uncertainty externalization.Examples include phrases such as “I’m not entirely sure,” which single-token counts cannot capture.
  • Uncertainty-expression measurement: SDPO substantially reduces annotated uncertainty expressions relative to both the baseline and GRPO, while GRPO shows greater variance than the baseline.The result is consistent with the token-level analysis.
  • Student–teacher divergence: Every epistemic-token variant exceeds the corpus-wide mean KL of 0.076, ranging from 1.9× for ‘Wait’ to 6.8× for ‘ Hmm’.Sentence-initial variants tend to have larger divergence, indicating strongest disagreement where the student initiates epistemic deliberation.

G Analysis of Changes in Model Internal Uncertainty under Self-Distillation

Richer conditioning lowers predictive uncertainty, and self-distillation transfers this lower-uncertainty behavior to students alongside suppressed epistemic verbalization. Prompting for uncertainty partly restores verbosity but does not eliminate performance degradation.

  • Internal uncertainty: Solution-guided generation reduces average token-level entropy by approximately 43%, while unguided epistemic tokens have entropy 3.5× above the overall average.Epistemic verbalization therefore tends to occur at positions of high predictive uncertainty.
  • Internal uncertainty: SDPO reduces token-level entropy while GRPO increases it, and this distinction persists throughout training.The result indicates that suppressed uncertainty expressions accompany a genuine reduction in internal uncertainty.
  • Prompt intervention: The uncertainty-encouraging prompt increases solution-guided response length from 1,873 to 3,098 tokens and epistemic expressions from 8.8 to 31.4.Both metrics nevertheless remain below unguided generation because the reference solution constrains reasoning verbosity.
  • Prompt intervention: The uncertainty-encouraging prompt produces slightly higher training scores and longer responses, but both prompted variants still show substantial performance degradation.The result suggests that prompting alone is insufficient to resolve uncertainty suppression in solution-guided generation.
  • Cross-model and related analyses: Under hybrid distillation, performance initially improves before steadily deteriorating as response length and epistemic-token counts decrease.The authors leave the causes of this dynamic, including reasoning changes or chat-template mismatch, for future work.
  • Cross-model and related analyses: SDPO lowers AIME24 acc@16 from 0.25 to 0.23 in one comparison, and degradation persists across Qwen3-1.7B, DeepSeek-R1-Distill-Qwen-7B, and OLMo-3-7B-Instruct.The reported degradation is 45.9% for Qwen3-1.7B, 12.1% for Qwen3-8B, and 40.7% for DeepSeek-R1-Distill-Qwen-7B; OLMo-3-7B-Instruct falls below its base model on OOD evaluation.

K.1 Unguided vs. Solution-Guided Generation

The response examples contrast exploratory, uncertainty-aware unguided reasoning with concise solution-guided reasoning. A concrete AIME24 example shows that checking an additional constraint distinguishes the correct GRPO solution from the incorrect SDPO solution.

  • Reasoning-style contrast: Unguided reasoning revisits assumptions, considers alternatives, and self-corrects, whereas solution-guided reasoning is more concise and direct.The paired responses address the same problem, illustrating different reasoning styles under unguided and solution-guided generation.
  • Reasoning-style contrast: The unguided rook-placement response explicitly explores uncertainty and alternative counting strategies before reaching the total 40,320.Its reasoning includes repeated checks and questions about the combinatorial interpretation.
  • Rook-placement example: The solution-guided rook response proceeds directly through matching counts and obtains 25,920 + 14,400 = 40,320.The calculation combines the white-square and black-square cases.
  • AIME24 example: For the AIME24 grid problem, GRPO derives the additional constraint a + b + c = 8 and correctly counts 45 solutions.The reference answer is 45.
  • AIME24 example: SDPO stops after deriving a + d = b + e = c + f = 9 and counts 1,000 cases without checking the row-sum equation.This misses the additional constraint a + b + c = 8 and produces the incorrect answer 1,000.
Loading 2603.24472v4…