PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking On-Policy Self-Distillation for Thinking Models

Simran Kaur, Narutatsu Ri, Yinghui He, Liam Fowl, Sanjeev Arora

arXiv:2607.05184v1cs.AIcs.LG

TL;DR

Thinking models can use long test-time deliberation for self-improvement, but it was unclear whether privileged-context self-distillation preserves that behavior. The paper evaluates privileged on-policy distillation and finds that it suppresses deliberative branching, degrading thinking-model performance especially at long rollout budgets.

  • Problem

    It remained unclear whether privileged-context self-distillation benefits thinking models when supervised trajectories are long deliberation traces.

  • Method

    The paper evaluates context-explicit on-policy distillation and diagnoses fork-versus-lock behavior and token-level signals along sampled thinking-model rollouts.

  • Results

    Privileged-context distillation degrades five thinking models, reverses unprivileged OPD gains, and most strongly harms long-budget behavior while suppressing forks and deliberation markers.

  • Takeaways & Limitations

    Privileged supervision can undermine thinking-model test-time search, so long-horizon distillation should control the teacher context and token-level learning signal.

  • Takeaways & Limitations

    The proposed fork-suppression explanation is not proven causal, the paper offers no solution, and experiments focus on verifiable mathematics.

Abstract

from arXiv · show

Self-distillation is a promising recipe for self-improvement in language models. In this setting, a model can serve as its own teacher when given privileged information, such as a solution to a math problem. This seems especially appealing for thinking models, which can use test-time reasoning to absorb the privileged information. Surprisingly, we show that privileged self-distillation degrades thinking models on long reasoning traces: across five Qwen3 and OLMo thinking models evaluated on AIME24, AIME25, and HMMT25, privileged-context distillation causes a relative drop of up to 17% in avg@16 accuracy. The degradation scales with the amount of privileged context withheld from the student and is most pronounced at long rollout budgets, where thinking models otherwise obtain their largest gains. This failure mode is not specific to self-distillation: on-policy distillation (OPD) improves thinking models, but privileged OPD reverses these gains. Our diagnostics link this failure mode to how privileged teacher context reshapes learning at high-entropy forking positions, where multiple continuations remain plausible and may lead to different reasoning paths. Privileged context lowers fork rates in thinking-model rollouts but not in instruction-model rollouts. This leads to an interesting dichotomy, where privileged context can help instruction-tuned models but hurts stronger thinking models. The effect is visible when the student begins a self-correction branch, where privileged OPD penalizes sampled reconsideration tokens that vanilla OPD supports. Thinking models trained with a privileged teacher produce fewer verification, backtracking, and hedging markers, even after length normalization. These findings indicate that self-distillation for strong thinking models requires attention to token-level signal, especially around correction and reasoning steps.

1 Introduction

Privileged-context on-policy self-distillation can degrade thinking models, especially at long rollout budgets, despite improving instruction-tuned models. The paper attributes this failure to fork suppression: privileged context reverses token-level learning signals that support exploration and deliberation at high-entropy decision points.

  • Introduction: OPSD helps instruction-tuned models more reliably than thinking models, whose long deliberation traces contain branching, verification, hedging, backtracking, and error recovery.Prior work mainly studied instruction-tuned models, short generation budgets, or rollouts with thinking disabled [Zhao et al., 2026, Shenfeld et al., 2026, Hübotter et al., 2026].
  • Introduction: Privileged-context on-policy self-distillation degrades thinking models, particularly at long rollout budgets, whereas vanilla OPD with a larger teacher improves them under the same short-budget setup.This shows that the harm is not explained by the short training completion budget alone.
  • Introduction: Across five OpenThoughts-trained Qwen3 and OLMo thinking models, full-solution privileged context causes degradation, while final-answer-only context causes milder damage.The harm increases with the amount of privileged context provided to the teacher.
  • Introduction: At high-entropy forking positions, privileged context appears to reverse the per-token distillation signal, suppressing branching, reconsideration, and uncertainty-marking moves.Thinking-model rollouts contain many such positions, motivating token-level mechanistic analysis [Bigelow et al., 2024, Lin et al., 2024, Vassoyan et al., 2025].
  • Introduction: The paper calls this phenomenon fork suppression: privileged-context self-distillation undermines the deliberative behaviors that make thinking models promising candidates for self-improvement.These behaviors include branching, verification, hedging, backtracking, and recovery from errors.

2 Experimental Setup

The experiments test whether privileged-context on-policy self-distillation preserves thinking models’ test-time search behavior across math reasoning and Countdown. They compare instruct and thinking models using long-budget evaluations on standard benchmarks and held-out Countdown data.

  • Method: The study minimizes a token-level divergence between teacher and student next-token distributions on student-sampled rollouts.The divergence may be forward KL, reverse KL, JSD, or another token-level distillation divergence.
  • Method: Vanilla OPD gives the teacher no privileged context, whereas privileged-context OPD supplies information such as a final answer or gold demonstration.In OPSD, teacher and student share the same architecture and typically start from the same model.
  • Models and comparisons: OPSD lowers avg@16 accuracy on AIME24, AIME25, and HMMT25 for all five evaluated thinking models across model families.The models are trained with privileged OpenThoughts solution context, and the Average column averages the three benchmarks.
  • Training data: Training uses 15k OpenThoughts math examples with full reference solutions as teacher context and 15k Countdown-CoT-20k examples with post-thinking solution suffixes.Countdown reserves 500 held-out examples for in-domain evaluation.
  • Models and comparisons: The comparisons span paired instruct and thinking Qwen3-4B [Yang et al., 2025] and OLMo-3-7B [Team Olmo et al., 2025] models plus multiple larger and smaller thinking models.Evaluation covers AIME 2024, AIME 2025, HMMT 2025, and held-out Countdown, using 16 samples per problem with a maximum length of 38,912 tokens.

3 Results

Privileged-context OPSD consistently harms thinking models, especially on long reasoning rollouts, while vanilla OPD and OPSD are more beneficial for instruction-tuned models. The degradation depends on thinking rollouts during training and increases with the amount of privileged teacher context.

  • Model family comparison: OPSD helps instruction models more reliably than thinking models: Countdown-trained instruct models improve from 0.527 to 0.559 and 0.474 to 0.494, while matched thinking models show mixed results.A self-revision pipeline similarly benefits instruction-tuned models after OPSD but hurts thinking models.
  • Overall results: All five evaluated thinking models lose avg@16 performance under OPSD, with drops ranging from 6.4 points for Qwen3-1.7B to 0.8 points for OLMo-7B-Think.Bootstrap intervals exclude zero for four of five comparisons, including all Qwen variants.
  • Training rollout format: When Qwen3-4B is trained on non-thinking rollouts, performance is preserved at 0.591 →0.590, whereas thinking-rollout training reduces it to 0.532.All rows are evaluated with thinking enabled.
  • Privileged versus vanilla distillation: Vanilla OPD improves Qwen3-1.7B from 0.372 to 0.392 avg@16, but privileged-context OPD reduces performance to 0.350 and OPSD with a full gold demonstration to 0.308.This isolates privileged teacher context as harmful rather than on-policy distillation itself.
  • Rollout budgets: At 4k–8k-token budgets, OPSD is comparable to or better than the base, but by 32k–38k tokens it matches or falls below the base and produces substantially shorter rollouts.OPSD therefore removes the gains thinking models otherwise obtain from longer rollouts.
  • Privileged-context amount: Full gold demonstrations cause larger long-budget reversals than final-answer-only context, while sparse context preserves long-budget behavior better.Full demonstrations also produce the largest short-budget gains and compress mean rollout length to roughly 0.8× the base at 32k tokens.

4 Analysis: Privileged Context Reduces Forking

Privileged context reshapes thinking-model rollouts by reducing high-entropy forking and increasing locally constrained continuations. This altered token-level signal suppresses self-correction and deliberation markers, although the diagnostics establish association rather than causality.

  • Fork–lock diagnostic: Dense privileged context monotonically lowers fork rates and raises lock rates in thinking rollouts, while instruction-style rollouts remain essentially flat.Fork positions represent high-entropy decisions with multiple plausible continuations, whereas lock positions are comparatively determined.
  • Token-level credit: Privileged scoring suppresses the self-correction cue But wait (−1.96, −4.97) and shifts positive credit toward the fluent continuation how can.This reversal appears when the trajectory is held fixed and only the teacher’s privileged information changes.
  • Deliberation markers: Gold-demonstration distillation lowers epistemic-token density to 0.850%, versus 1.080% →1.074% under vanilla OPD, including wait at 3.85 →2.54 per 1,000 tokens.The same suppression appears in pre-sampling probability mass, especially for wait, recall, altern, and hmm.
  • Deliberation markers: Across paired thinking-model rollouts, OPSD reduces verification, backtracking, and hedging markers after length normalization; verification falls from 1.63 to 1.35 per 1,000 tokens.The paired evaluation covers five thinking models and 7,200 rollouts across AIME24, AIME25, and HMMT25.
  • Interpretation and limitation: These diagnostics identify a consistent behavioral pattern associated with accuracy degradation but do not establish a causal explanation.The pattern includes lower fork rates, self-correction credit reversals, and fewer deliberation markers under privileged-context distillation.

5 Related Work

This work studies privileged-context on-policy self-distillation, where a self-teacher receives information unavailable to the student. Prior methods have reported gains in this setting, including OPSD, SDFT, and SDPO.

  • 5 Related Work: Privileged-context on-policy self-distillation gives the self-teacher answers, demonstrations, or feedback unavailable to the student.Self-distillation broadly uses a teacher derived from the student, often sharing its architecture or checkpoint.
  • 5 Related Work: Prior work reported gains from OPSD, SDFT [Shenfeld et al., 2026], and SDPO [Hübotter et al., 2026].

6 Discussion

Privileged supervision interacts delicately with test-time search in thinking models: privileged-context OPD can reduce long-budget gains even when unprivileged OPD improves the same student. The degradation also depends on the teacher-side context format, with short final-answer-only context preserving long-budget behavior better than a full gold demonstration.

  • 6 Discussion: Privileged-context OPD reduces thinking models’ long-budget gains, despite closely related unprivileged OPD improving the same student.Token-level analyses plausibly attribute this effect to changes in the learning signal at forking positions when the teacher sees unavailable information.
  • 6 Discussion: Long-rollout degradation depends on privileged teacher-side context: short final-answer-only context preserves long-budget behavior better than a full gold demonstration.This distinction is relevant to OPSD-style training for long-horizon agents and emerging training recipes using targeted textual feedback.

7 Limitations … B.4 OPSD Training

The appendix qualifies the paper’s primarily negative findings by noting that its failure analysis is not fully causal, while documenting related work, datasets, assets, prompt constructions, and OPSD training procedures. It specifies how students and privileged teachers are prompted and how on-policy token-level distillation is trained.

  • 7 Limitations: The failure analysis is not perfectly isolated or proven causal, although the authors link OPSD’s suppression of forking behavior to established work on reasoning tokens.The authors characterize their explanation as a convincing hypothesis rather than a definitive causal account.
  • B Experimental Details: The appendix records the data, training, and evaluation settings, using Table 9 defaults and Table 11 generation and grading protocols unless otherwise stated.Table 8 lists existing datasets, model checkpoints, and software assets used for training, evaluation, or implementation.
  • B.1 Data: OpenThoughts math and Countdown each contribute a 15K training subset, while Countdown additionally reserves 500 held-out evaluation examples.OpenThoughts math is cleaned and reformatted before training; its privileged teacher context is taken from the processed data.
  • B.2 Existing Asset Licenses: The work uses third-party datasets, model checkpoints, and software assets for training, evaluation, or implementation without redistributing them.The appendix summarizes these assets in Table 8.
  • B.3 Prompt Templates and Privileged Context Examples: Standard OPSD prompts the student with only the problem, whereas the privileged teacher receives the same problem plus an example response.Dense teacher context contains a full reference solution, while sparse context contains only the boxed final answer; the conciseness-control teacher receives no gold context.
  • B.4 OPSD Training: For each example, the trainer samples completions on-policy and minimizes token-level divergence between a privileged teacher and student on the sampled completion tokens.Both models start from the same base checkpoint, but only the teacher receives privileged supervision.

B.5 Evaluation

Evaluation uses 16 sampled rollouts per problem on AIME24, AIME25, and HMMT25, with the same protocol on Countdown when reported. Results distinguish avg@16 from unbiased pass@k and include deliberation-marker analyses for paired base and OPSD rollouts.

  • Evaluation protocol: Evaluation samples 16 rollouts per problem across 30 problems for AIME24, AIME25, and HMMT25, using up to 38,912 generated tokens.Each benchmark therefore contributes 480 generated rows; Countdown uses 16 samples on its 500-example held-out split when reported.
  • Metrics: avg@16 averages correctness across 16 sampled rollouts and problems, whereas pass@k uses an unbiased estimator and is reported only for k ≤ n.avg@16 is distinct from unbiased pass@16 and corresponds to empirical single-sample correctness under the evaluation sampling distribution.
  • Deliberation-marker analysis: Deliberation-marker analysis compares paired base and OPSD rollouts using raw and length-normalized marker counts with clustered-bootstrap confidence intervals.Normalized counts are reported per 1,000 generated tokens, with deltas defined as OPSD minus base.

B.6 Deliberation Marker Analysis · C Accuracy Confidence Intervals and Full Pass@k Results · C.1 Paired Bootstrap Confidence Intervals

The analysis examines explicit deliberation markers in paired base and OPSD reasoning traces and quantifies avg@16 uncertainty with clustered paired bootstrap intervals. It reports that OPSD is more reliable for instruction models, while distillation and privileged teacher context degrade thinking-model performance.

  • B.6 Deliberation Marker Analysis: The deliberation analysis compares five thinking models across AIME24, AIME25, and HMMT25 using 7,200 paired base and OPSD rollouts.Each benchmark contributes 30 problems with 16 samples per problem for each model.
  • B.6 Deliberation Marker Analysis: Responses are scored with a fixed case-insensitive lexicon covering verification, backtracking, and hedging markers, reported both as raw counts and per 1,000 tokens.Examples include “check,” “wait,” and “maybe.”
  • B.6 Deliberation Marker Analysis: OPSD-minus-base marker differences use clustered bootstrap intervals, while mean response length falls from 19,395 to 15,391 tokens after OPSD.The lexical counts are treated as proxies for explicit deliberation markers, not direct measures of latent uncertainty or confidence.
  • C.1 Paired Bootstrap Confidence Intervals: The confidence intervals resample evaluation problems within each benchmark while preserving paired method measurements, then average benchmark-level deltas.Tables 14–16 report paired 95% bootstrap intervals for avg@16 deltas from Tables 2, 1, and 4.
  • C.1 Paired Bootstrap Confidence Intervals: OPSD helps instruction models more reliably than thinking models on Countdown, according to paired bootstrap 95% confidence intervals for avg@16 deltas.This result is reported in Table 14.
  • C.1 Paired Bootstrap Confidence Intervals: OPSD degrades thinking models across model families, with paired bootstrap 95% confidence intervals reported for avg@16 deltas.This result is reported in Table 15.
  • C.1 Paired Bootstrap Confidence Intervals: Privileged teacher context, rather than on-policy distillation itself, degrades thinking-model performance under the reported avg@16 confidence-interval analysis.This result is reported in Table 16.

C.2 Full Pass@k Results · D Fork/Lock Token Measurement

The appendix provides full pass@1/pass@16 companion tables for the main benchmark results and details fork/lock-token measurements under multiple teacher-context conditions. The measurement protocol uses fixed student traces, entropy- and support-based classifications, and per-trace rate aggregation.

  • C.2 Full Pass@k Results: The appendix pass@1 values match the main tables’ avg@16 values because both average correctness over the same 16 sampled rollouts, whereas pass@16 estimates solving at least once among those rollouts.The supplied passage truncates before completing the pass@16 definition.
  • C.2 Full Pass@k Results: Full-format tables report pass@1 / pass@16 results for Countdown-trained think-vs-instruct models, OpenThoughts-trained thinking models, and Qwen3-1.7B OPD comparisons.The tables cover the benchmark sets and averages specified for each comparison, but the supplied passages do not provide cell values.
  • D Fork/Lock Token Measurement: Fork- and lock-like positions are measured from teacher next-token distributions on fixed student traces generated from 60 OpenMathReasoning prompts.Each trace is evaluated at every generated token under base, sparse, and dense teacher-context conditions.
  • D Fork/Lock Token Measurement: The final SRT runs use K = 3 retained next-token log-probabilities, 3072-token limits for base and sparse conditions, and shorter dense-condition reference and student traces.All six OPSD/OPD-by-condition cells are completed with 60 traces per cell.
  • D Fork/Lock Token Measurement: Entropy thresholding labels positions with normalized retained-distribution entropy Hnorm ≥ 0.60 as fork tokens and all remaining positions as neutral.The entropy is normalized over the retained top-K support.
  • D Fork/Lock Token Measurement: A support-aware approximation labels positions lock-like when retained support is sharply concentrated and fork-like when multiple retained tokens remain competitive.Top-p truncation uses p = 0.8; tail tokens are not treated as forks, and ambiguous positions remain neutral.
  • D Fork/Lock Token Measurement: Fork, lock, and neutral rates are computed per trace and condition, then visualized as OPSD/OPD-separated boxplots across base, sparse, and dense settings.Boxes summarize 60 traces, jittered points show individual traces, and diamonds indicate means.

E OPD Ablations

Token-masked OPD variants still improve the base thinking model, but neither clearly outperforms the other, and they do not reproduce gold-demonstration degradation. Their near-vanilla epistemic-token behavior contrasts with privileged-context training, which suppresses revision-related probabilities and realized marker density.

  • Token-masked OPD: Epistemic-token OPD and random-fraction OPD perform similarly, so the ablation does not clearly identify one masking strategy as superior.Epistemic markers may proxy for forking behavior, but branch-relevant decisions also occur elsewhere in the response.
  • Token-level diagnostics: Sparse-loss OPD controls leave epistemic-marker probabilities nearly unchanged relative to vanilla OPD, unlike gold-demonstration conditioning.The contrast appears in both aggregate marker probability mass and average log-probability.
  • Token-level diagnostics: Gold-demonstration conditioning produces the largest decrease in realized epistemic-token density, including clear reductions for representative revision markers.Epistemic-token and random-fraction OPD only slightly reduce aggregate marker density relative to the base and vanilla OPD.
  • Model-type comparison: OPSD helps an instruction-tuned Qwen3-4B-Instruct model but hurts the thinking-model Qwen3-4B across the reported benchmark comparison.The comparison covers the base model, OPSD alone, self-revision training alone, and the combined SRT+OPSD pipeline using avg@8 accuracy.

F SD-Zero Self-Revision Pipeline · G Additional Budget-Curve Figures

The SD-Zero pipeline shows that self-revision training helps both instruction-tuned and thinking models, whereas subsequent privileged on-policy self-distillation reverses the thinking-model gain. Additional budget curves isolate pass-rate and response-length effects across models, contexts, rollout budgets, and conciseness prompting.

  • F SD-Zero Self-Revision Pipeline: SD-Zero first trains self-revision with reward feedback and then distills the reviser into the generator through on-policy self-distillation.The comparison includes the base model, self-revision training alone, and the full SRT+OPSD pipeline on Qwen3-4B-Instruct.
  • F SD-Zero Self-Revision Pipeline: On Qwen3-4B, SRT improves instruction-tuned accuracy by 11.3 points and OPSD adds 2.6, while on the thinking model OPSD reverses SRT’s gain by 3.3 points.The results identify the subsequent OPSD stage, rather than self-revision training, as the source of degradation for the thinking model.
  • G Additional Budget-Curve Figures: Figure 6 separates pass-rate budget curves across five thinking models, three benchmarks, and rollout budgets from 4k to 38k tokens.It compares base thinking models with OPSD using full gold-demonstration context or final-answer-only privileged context; dense demonstrations tend to yield larger short-budget gains.
  • G Additional Budget-Curve Figures: Figure 7 shows response-token budget curves for the same models, benchmarks, and rollout budgets, revealing where OPSD compresses long thinking rollouts.Solid lines report mean response length and dashed lines median response length across the compared contexts.
  • G Additional Budget-Curve Figures: Figure 8 shows that gold-demonstration OPD shortens responses, whereas vanilla OPD preserves the base length profile for a Qwen3-1.7B student with a Qwen3-8B teacher.The comparison uses 4,096-token training completions and generation caps from 4,096 to 38,912 tokens, with pass@1 and pass@16 curves.
  • G Additional Budget-Curve Figures: A CRISP-style conciseness prompt compresses Qwen3-8B responses but does not recover long-budget gains, despite using no gold context [Sang et al., 2026].CRISP conditions the teacher on conciseness rather than a gold answer or reference solution, while still providing token-level supervision across the rollout.
Loading 2607.05184v1…