PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision

Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, Sanjeev Arora

arXiv:2604.12002v2cs.CL

TL;DR

Post-training with binary rewards is broadly applicable but sparsely supervises reasoning, while dense distillation typically requires external teachers or high-quality demonstrations. SD-ZERO uses a generator-reviser model and on-policy self-distillation to transform rewards into token-level supervision, achieving consistent 10%+ gains across two model families and eight benchmarks while outperforming same-budget baselines. Its scope remains focused on concise instruct models and verifiable domains.

  • Problem

    Binary rewards lack information about intermediate reasoning, while dense distillation generally requires external teachers or high-quality demonstrations.

  • Method

    SD-ZERO trains one model as a generator and reward-conditioned reviser, then distills the reviser’s token distributions into the generator using on-policy responses.

  • Results

    Consistent 10%+ gains over base models across two model families and eight benchmarks outperform all baselines under the same training sample budget.

  • Takeaways & Limitations

    SD-ZERO transforms sparse binary outcome rewards into dense token-level supervision without an external teacher or high-quality demonstrations.

  • Takeaways & Limitations

    The study focuses on concise instruct models and verifiable domains; extending SD-ZERO to thinking models and non-verifiable rewards remains open.

Abstract

from arXiv · show

Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training. Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations. Collecting such supervision can be costly or unavailable. We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations. SD-Zero trains a single model to play two roles: a Generator, which produces an initial response, and a Reviser, which conditions on that response and its binary reward to produce an improved response. We then perform on-policy self-distillation to distill the reviser into the generator, using the reviser's token distributions conditioned on the generator's response and its reward as supervision. In effect, SD-Zero trains the model to transform binary rewards into dense token-level self-supervision. On math and code reasoning benchmarks with Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SD-Zero improves performance by at least 10% over the base models and outperforms strong baselines, including Rejection Fine-Tuning (RFT), GRPO, and Self-Distillation Fine-Tuning (SDFT), under the same question set and training sample budget. Extensive ablation studies show two novel characteristics of our proposed algorithm: (a) token-level self-localization, where the reviser can identify the key tokens that need to be revised in the generator's response based on reward, and (b) iterative self-evolution, where the improving ability to revise answers can be distilled back into generation performance with regular teacher synchronization. Code: https://github.com/princeton-pli/Self-Distillation-Zero.

1 Introduction

RLVR offers broadly applicable binary rewards but sparse supervision, while distillation needs external teachers or high-quality demonstrations. SD-ZERO addresses this gap by using self-revision and self-distillation to convert outcome rewards into dense supervision.

  • Binary rewards indicate final-answer correctness but not which intermediate reasoning steps are sound, making training expensive.
  • Existing on-policy distillation methods rely on external stronger teachers, while newer self-distillation methods still require high-quality demonstrations.
  • SD-ZERO uses one model as both generator and reviser, conditioning revision on the model’s response and its reward.
  • SD-ZERO first retains successful self-revision traces, then distills the reviser’s token-level behavior into the generator on policy.
  • 10.5% and 10.4% total gains over base models are reported for Qwen3-4B-Instruct and Olmo-3-7B-Instruct, respectively, across math and code reasoning benchmarks.

2 SD-ZERO: Turning Binary Rewards into Dense Self-Supervision

SD-ZERO trains a single model to generate responses, revise them using binary outcome feedback, and distill revision behavior back into generation. Its two phases use self-revision traces followed by on-policy token-level distillation.

  • Binary reward r(y, a) equals 1 when the extracted final answer matches the ground truth and 0 otherwise; gold solutions are not assumed.
  • The single model serves as generator and reviser, with Phase 1 training both roles and Phase 2 distilling revision feedback into generator responses.
  • Phase 1: Self-Revision Training: Phase 1 samples multiple initial responses, conditions revision on correctness, and retains corrected or rephrased outputs for self-revision training.
  • Phase 1: Self-Revision Training: The SRT objective combines revision loss with generation loss, preserving ordinary generation while training self-revision behavior.
  • Phase 2: Self-Distillation: Phase 2 freezes the SRT reviser, samples on-policy generator responses, and trains the generator to match the reviser’s token distributions via KL divergence.
  • Evaluation: Table 1 compares compute-equalized methods on math and code benchmarks using avg@8 across Qwen3-4B-Instruct and Olmo-3-7B-Instruct.

3 Experiments

Experiments show that SRT improves self-revision and sample-efficiently outperforms standard baselines, while Self-Distillation further strengthens performance, reduces response length, and enables continued self-evolution through teacher synchronization.

  • 3 Experiments: SRT improves average accuracy by 7.8% for Qwen3-4B-Instruct and 9.2% for Olmo-3-7B-Instruct using 6K self-generated revision traces.This outperforms SFT and RFT trained on 15K examples.
  • 3 Experiments: SRT increases Qwen3-4B-Instruct’s Generate-then-Revise gain from 1.1% to 5.0% on 1K AIME24 questions.The revisions are shorter than the model’s first attempts, indicating more targeted revision behavior.
  • 3 Experiments: Self-Distillation adds 2.7% for Qwen3-4B-Instruct and 1.2% for Olmo-3-7B-Instruct beyond SRT, reaching total gains of +10.5% and +10.4% over base models.The phase improves code and HMMT25 performance for Qwen and mainly math performance for Olmo.
  • 3 Experiments: SD-ZERO generates half as many tokens as the SRT model and fewer than all baselines while achieving the strongest overall performance.Self-Distillation internalizes revision behavior into more proactive generator responses.
  • 3 Experiments: SD-ZERO outperforms GRPO and SDFT by at least 4.8% on average across benchmarks under a matched generation budget.It uses only a scalar reward on the first attempt, whereas SDFT requires gold solutions and GRPO samples groups of responses.
  • 3 Experiments: After teacher synchronization, a second Self-Distillation phase adds at least 3 percentage points without signs of saturation.The model can continue self-evolving using the initial attempt and its binary reward in the teacher’s context.

4 Understanding How SD-ZERO Improves Reasoning

SD-ZERO improves reasoning by localizing token-level errors, distilling revision behavior into shorter generation, and relying on complementary training phases.

  • 4.1 Token-Level Self-Localization: Token-level self-localization concentrates reviser feedback on a small subset of tokens, especially in incorrect responses.For correct responses, the KL signal is more uniform; for incorrect responses, it identifies localized errors.
  • 4.1 Token-Level Self-Localization: The reviser both penalizes faulty reasoning and redirects probability toward plausible alternatives.In one example, faulty symmetry-based reasoning receives positive KL while correct coordinate-based reasoning receives negative KL.
  • 4.2 Iterative Self-Evolution: Self-Distillation reduces generation length by roughly 2× relative to SRT while improving performance beyond SRT.The distilled generator anticipates pitfalls proactively rather than explicitly backtracking.
  • 4.2 Iterative Self-Evolution: During Self-Distillation, response length and self-revision keyword frequency fall steadily while accuracy increases.SRT first raises both statistics as the model learns explicit self-correction; Self-Distillation then internalizes this behavior more concisely.
  • 4.3 Ablations on SD-ZERO Design: The SRT loss terms are complementary: revision training elicits self-revision, while generation training transfers it to stronger generation.Removing either term weakens the corresponding capability and fails to match full SRT.
  • 4.3 Ablations on SD-ZERO Design: SRT is necessary before Self-Distillation, while allocating more data to Self-Distillation yields the best final performance.Phase-2-only training produces only marginal gains, whereas additional post-SRT data is used more effectively to refine generation.

5 Discussion

SD-ZERO transforms binary correctness rewards into dense self-supervision through two-phase self-revision and self-distillation, achieving broad gains while remaining bounded to verifiable domains and short-response models.

  • 5 Discussion: Across two model families and eight benchmarks, SD-ZERO yields consistent 10%+ gains over base models and outperforms baselines at the same sample budget.The study covers math and code reasoning benchmarks and reports comparisons under matched training questions and sample budgets.
  • 5 Discussion: The study focuses on instruct models producing short, concise responses and leaves extension to long thinking models as an open direction.Long exploratory responses make it difficult to distinguish productive false starts from genuine errors and assign credit beyond local token decisions.
  • 5 Discussion: SD-ZERO is evaluated in verifiable domains such as math and coding; extending it to domains without verifiable rewards remains an open problem.The paper suggests meta-cognitive signals such as consistency or self-correction as a possible future direction.
  • 5 Discussion: Unlike on-policy distillation and many self-distillation methods, SD-ZERO requires only binary correctness signals rather than an external teacher or high-quality demonstrations.The method generates its own supervision through self-revision.
  • 5 Discussion: SRT retains incorrect attempts for revision-based learning, contrasting with self-training approaches that discard incorrect reasoning entirely.This connects SD-ZERO’s self-revision dataset to prior rationale bootstrapping methods.
  • 5 Discussion: The reviser’s per-token KL signal is functionally analogous to process reward models because both localize supervision over intermediate reasoning.SD-ZERO differs by deriving the signal from self-revision rather than requiring step-level annotations or separate reward-model training.
  • 5 Discussion: SD-ZERO uses a single model as generator and reviser, first eliciting self-revision and then distilling it into compact generation.Phase 1 trains on verified successful revisions; Phase 2 converts outcome-level reward into token-level supervision.

B.2 Comparison to Existing Methods

SD-ZERO combines on-policy sampling, self-distillation, and teacher conditioning on incorrect attempts to convert binary rewards into targeted token-level supervision without gold reasoning traces.

  • SD-ZERO is the only compared method combining on-policy training, self-distillation, and teacher conditioning on incorrect student attempts.
  • SD-ZERO uses 15K training examples separately from OpenR1-Math and Codeforces, split across SRT and Self-Distillation phases.
  • SRT curates 6K traces by sampling initial responses, verifying binary rewards, generating revisions, and retaining successful revisions.
  • The Self-Distillation phase uses an additional 9K question-answer pairs to provide token-level supervision over generator responses.

C.2 Comparing Sampling Budgets

SD-ZERO uses a comparable overall sampling-token budget to GRPO and RFT while requiring fewer total generations, with separate budgets for revision training and self-distillation.

  • RFT and GRPO each use 60K generations from 15K questions with four response attempts per question.
  • 49K total generations comprise 40K in SRT and 9K in Self-Distillation.
  • Approximately 225M tokens are generated by SD-ZERO, versus roughly 222M for each of RFT and GRPO.SD-ZERO generates approximately 148M tokens during SRT and at most 76.5M during Self-Distillation.
  • SD-ZERO’s SRT collection uses 10K initial responses and 30K revisions, totaling 148M tokens.
  • SRT training consumes 66M tokens, while Self-Distillation consumes at most 153M tokens across student and teacher forward passes.

D.1 Pass@8 Performance on Math Benchmarks

Across Pass@8 math evaluation, SRT matches or exceeds strong baselines and SD-ZERO achieves the strongest overall performance; larger GRPO rollout budgets do not close the gap.

  • SRT matches or outperforms strong baselines across Qwen3-4B-Instruct and Olmo-3-7B-Instruct, while SD-ZERO achieves the best overall average performance.
  • Pass@8 gains suggest SD-ZERO is not merely concentrating outputs around a narrow set of solutions.
  • Final-answer-only SDFT remains close to the base model and falls far behind SD-ZERO across all evaluated math benchmarks.
  • Increasing GRPO from 4 to 8 generations per question yields only marginal improvements and can slightly reduce performance in some settings.

E.1 Self-Revision Keyword Analysis

The SRT ablations show that revision and generation objectives are complementary, SRT is needed before self-distillation, and the phase split creates a performance trade-off.

  • Loss-term ablation: Both SRT loss terms are necessary: neither Lrevision nor Lgeneration alone recovers the full SRT effect.
  • Loss-term ablation: Lgeneration alone reduces correction rate from 15.0% to 7.2%, while Lrevision alone reaches 12.1% but substantially lowers generation quality.
  • Phase ablation: Removing SRT before Self-Distillation leaves correction rate at 2.6%, nearly unchanged from the base model’s 2.7%.
  • Phase ablation: SRT raises correction rate to 15.0%, and full SD-ZERO raises it further to 16.7%.
  • Data-split ablation: Allocating more data to SRT slightly improves the SRT model, but the best final SD-ZERO accuracy comes from assigning more data to Self-Distillation.

F SDFT with Thinking Models

Extending self-distillation to thinking models is non-trivial: enabling thinking during SDFT training degraded the base model across all tested benchmarks, whereas disabling it largely preserved accuracy.

  • Motivation: Thinking models generate long, exploratory chains of thought, making self-distillation a natural but non-trivial extension.
  • Setup: The experiment uses Qwen3-4B with thinking enabled or disabled during SDFT training, while evaluation always enables thinking.The sole experimental variable is the enable_thinking flag.
  • Results: Enabling thinking during SDFT training degraded the base model on every tested benchmark.All models were evaluated with thinking enabled, so the comparison isolates the student’s training-time thinking mode.
  • Results: −9.8 points on AIME24, −10.8 on AIME25, and −8.3 on HMMT25 resulted from enabling thinking during training.
  • Results: Disabling thinking during training largely preserved the base model’s accuracy and slightly improved it on HMMT25.

G.1 Effect of Training Data Source

Training-data source strongly affects SFT performance: on-policy data and self-revision outperform off-policy or first-attempt alternatives, while correctness filtering is essential for revision traces.

  • Setup: The comparison samples multiple solutions per question from 8K original SFT questions and optionally filters them for correctness.
  • Setup: The evaluated sources include on-policy GRPO data and self-revision variants, including unfiltered and filtered self-revision.
  • Findings: On-policy data substantially outperforms off-policy teacher data despite comparable data sizes and correctness filtering.SFT on GRPO-generated data reaches approximately 0.88, versus 0.605 for LLaMA-70B and 0.617 for GPT-4o data.
  • Findings: Filtered self-revision reaches pass@1=0.630, exceeding correctness-filtered first attempts at pass@1=0.552.Both sources are on-policy, isolating the value of the revision step itself.
  • Findings: Filtering raises self-revision performance from 0.529 unfiltered to 0.630 filtered, a 19% relative gain.The unfiltered variant underperforms the base model’s first attempts, indicating that noisy revision traces hurt.

G.2 Self-Revision as Initialization for RL

Self-revised targets provide a stronger initialization for RL on out-of-distribution benchmarks than teacher-generated targets or standalone GRPO, although the comparison is conservative and the extension beyond verifiable rewards remains future work.

  • Results: Countdown (A′)+GRPO achieves pass@128 of 0.40 on AIME24 and 0.47 on AIME25, exceeding Countdown (A)+GRPO at 0.23 and 0.40.Standalone GRPO reaches 0.30 on AIME24 and 0.40 on AIME25; in-distribution Countdown performance is identical across GRPO variants at 0.91/0.98.
  • Results: Self-revised targets provide a better RL initialization on OOD benchmarks than LLaMA-70B-generated targets.The evaluation compares Countdown training with teacher-generated answers A or self-revised answers A′, optionally followed by GRPO.
  • Caveat: The comparison is conservative because neither target set is correctness-filtered and A uses off-policy answers.The authors state that gains would likely be larger with filtering.
  • Limitation: Extending self-revision training to settings without verifiable rewards remains future work because correctness filtering is unavailable there.
Loading 2604.12002v2…