PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Self-Improvement via On-Policy Distillation for Reasoning

Shangjian Yin, Zehao Zhao, Kavosh Asadi, Rui Liu, Yuchen Lu, Shike Mei, Hang Cui, Luke Simon, Zhouxing Shi, Hamed Firooz

arXiv:2609.30652v1cs.CL

TL;DR

A frozen privileged teacher may become misaligned as its student learns stronger revision behavior. The paper combines Dynamic Co-Evolution, which refreshes the privileged teacher from each updated checkpoint, with Self-Refined Concise Learning on shorter verified rewrites. DCE+SRCL reaches 65.97% Average@12 on Qwen3-8B, compared with 30.35% for OPSD.

  • Problem

    A privileged teacher frozen at the initial checkpoint cannot acquire revision behaviors emerging during student post-training, so its guidance may become increasingly misaligned with later student trajectories.

    Source: §1 · p. 2

  • Method

    DCE refreshes a detached, gold-conditioned privileged teacher from each updated checkpoint, while SRCL trains on shorter, verified refinements of the same on-policy response.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    65.97% Average@12: DCE+SRCL on Qwen3-8B, compared with 30.35% for OPSD.

    Source: §5.1 · p. 8

  • Takeaways & Limitations

    Together, DCE and SRCL deliver stronger and more token-efficient recursive self-improvement across model scales.

    Source: §6 · p. 13

Abstract

from arXiv · show

On-policy distillation (OPD) trains a student model by having it generate trajectories, then matching its next-token predictions with an external teacher's next-token predictions. This provides dense, token-level supervision to the student. On-policy self-distillation (OPSD) eliminates the need for the external teacher. Specifically, a second frozen copy of the student model, now given the ground truth in its context, serves as the teacher. The student model only receives the problem and learns to mimic the privileged teacher model, while the teacher remains frozen throughout training. Previous work showed that freezing the teacher is useful for training stability, but we argue that this can prevent the teacher from incorporating the improvements learned by the student during training. Our primary contribution is to address this limitation with a recursive framework built around two complementary components. First, we let the privileged teacher co-evolve with the student so that revision learned in one round can guide the next, a process we refer to as Dynamic Co-Evolution (DCE). Second, because stronger revision can also make responses too verbose and self-critical, we additionally train on shorter, verified rewrites of the model's own on-policy responses. We call this complementary objective Self-Refined Concise Learning (SRCL). Overall, our comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks. Specifically, on Qwen3-8B, DCE+SRCL reaches 65.97% Average@12, outperforming OPSD by 35.62 percentage points while reducing mean output length by 7.80% relative to DCE alone.

1 Introduction

The introduction motivates token-level guidance that can teach revision, noting that outcome rewards do not identify where reasoning failed and that a frozen privileged teacher can become misaligned with an evolving student. It proposes Dynamic Co-Evolution (DCE) and Self-Refined Concise Learning (SRCL) to strengthen revision while controlling response length.

1 Introduction

  • Outcome rewards provide sequence-level correctness signals but do not indicate where reasoning went wrong or how it should be revised.Process supervision offers more local feedback but generally requires step-level annotations or a separately trained verifier.Source: §1 · p. 1
  • Fixed-trace probes show the frozen teacher favors termination on incorrect trajectories, while the student grows more likely to reflect during training.The frozen teacher assigns an average 94.5% probability to EOS at incorrect response endpoints and 22.0% to observed reflection cues; the passages also report the student’s growing likelihood of reflection.Source: §1 · p. 2
  • SRCL trains on shorter, answer-verified rewrites of the same on-policy responses to complement DCE’s stronger revision and limit unnecessary token cost.The introduction identifies more frequent checking and backtracking as behaviors that can make reasoning longer.Source: §1 · p. 2
  • DCE improves Average@12 over OPSD by 35.41, 37.15, and 12.98 percentage points at 8B, 4B, and 1.7B, respectively.Each updated checkpoint initializes both the next student and a detached, gold-conditioned privileged teacher.Source: §1 · p. 2
  • 35.07% Average@12: DCE+SRCL beats the OPSD test-time-scaling control by 6.60 percentage points under an 8K evaluation budget, with approximately 692 fewer tokens per response.Mean output length is 7,500 tokens per response.Source: §1 · p. 3

2 Related Work

Related work describes limits of outcome-level supervision for error repair, contrasts on-policy distillation with privileged self-distillation, and situates concise self-refinement among self-training and test-time scaling approaches.

2.1 Reflection and Verification in Reasoning Post-Training

  • Outcome-based post-training can elicit reflection-like behavior, but rewards score completed responses rather than supervising the local transition that identifies and repairs a specific error.Prior findings suggest reflection may be a pre-existing behavioral prior, and reflection does not reliably correct errors without external feedback.Source: §2.1 · p. 3

2.2 On-Policy and Privileged Self-Distillation

  • OPSD uses on-policy student responses and a frozen copy of the initial checkpoint as a privileged teacher that also receives a verified solution.The teacher supplies token-level guidance along student-generated prefixes; the standard setup keeps it fixed at θ0 and gives it no gradients.Source: §2.2 · pp. 3–4

2.3 Self-Refinement and Efficient Reasoning

  • SRCL retains only shorter, naturally terminated, structurally valid, answer-correct rewrites of the model’s own on-policy responses, complementing DCE’s revision guidance.Unlike test-time scaling, SRCL changes the model during post-training to preserve correct reasoning with fewer tokens in a single rollout.Source: §2.3 · p. 4

3 Method

The method couples recursive gold-conditioned distillation with accepted concise self-refinements, using each joint update to initialize both roles in the next round.

3.1 Dynamic Co-Evolution

  • At each round, the student and privileged teacher score the same on-policy prefixes; the teacher also sees the verified solution, and the updated checkpoint initializes both roles next round.The teacher's gradients are stopped, while its checkpoint evolves between rounds.Source: §3.1 · pp. 4–5

3.2 Self-Refined Concise Learning

  • SRCL asks the same checkpoint to rewrite each response without the verified solution, retaining only shorter, naturally terminating, structurally valid rewrites with the correct answer.Source: §3.2 · p. 6
  • Accepted rewrites become token-level supervised targets; if a minibatch has no accepted refinement, the SRCL loss is omitted and the model updates using DCE alone.Source: §3.2 · p. 6

3.3 Joint Optimization

  • The joint objective combines DCE guidance with SRCL targets, with λG and λS controlling their relative contributions.Source: §3.3 · p. 6

4 Experimental Setup

The experiments evaluate Qwen3 models and a cross-family Gemma study on competition-level mathematics benchmarks, using training baselines and an inference-time control.

4.1 Models, Data, and Metrics

  • Evaluation samples 12 responses for each of 30 problems from AIME 2024, AIME 2025, AIME 2026, and HMMT February 2025.The training data comprises 14,717 OpenThoughts mathematical-reasoning problems; the study includes Qwen3-1.7B, 4B, 8B, and 14B, plus Gemma-4-12B-IT in a cross-family study.Source: §4.1 · pp. 6–7

4.2 Baselines and Controls

  • The training comparisons include Base, SFT, GRPO, and OPSD; OPSD-TTS tests whether extending generation alone accounts for DCE+SRCL gains.OPSD-TTS resumes early-terminating OPSD responses to an exact 8K or 16K token budget without changing model parameters.Source: §4.2 · p. 7

4.3 Training Protocol

  • Qwen3 training uses AdamW, bfloat16, a 5×10^-6 learning rate, rank-128 LoRA, and one on-policy response per training example.The DCE loss weight is 5×10^4 for all Qwen3 models; checkpoints are evaluated at available steps among 10, 20, 30, 50, 100, 150, and 200.Source: §4.3 · pp. 7–8

5 Results

Across model scales and evaluation settings, DCE+SRCL improves reported accuracy, often with shorter outputs, and its learned revision behavior remains useful under tighter budgets.

5.1 Main Experimental Results

  • 65.97% Average@12 on Qwen3-8B and 61.88% on Qwen3-4B make DCE+SRCL the leading reported non-thinking results, versus OPSD's 30.35% and 22.85%.At Qwen3-1.7B, accuracy rises from 23.33% to 26.88%, but mean output length also rises from 12,600 to 19,504 tokens.Source: Table 1 · p. 8, §5.1 · pp. 8–9

5.2 Does Recursive Improvement Transfer Beyond Qwen3?

  • On Gemma-4-12B-IT, DCE and DCE+SRCL reach 62.01% and 63.61% Average@12, respectively, outperforming all non-DCE comparisons.SRCL adds 1.60 percentage points while reducing mean output length from 8,812 to 8,526 tokens.Source: §5.2 · pp. 9–10, Table 2 · p. 9

5.3 What Happens During Recursive Self-Improvement?

  • Across three incorrect-trajectory cohorts, endpoint p(EOS) falls while the probability of the observed reflection cue rises for both Student and Teacher by step 200.Macro-averaged Student/Teacher p(EOS) changes from 92.6%/90.4% at initialization to 26.3%/41.3%; p(rt) changes from 30.1%/32.8% to 77.0%/77.4%.Source: §5.3 · p. 10, Figure 4 · p. 10

5.4 How Do Different Loss Weights Affect Performance?; 5.5 How Does the Guidance Objective Influence Performance?

  • Across tested loss weights, accuracy is locally robust but non-monotonic; Forward KL outperforms Reverse KL and JSD, while SRCL alone collapses without DCE.Forward KL reaches 65.97% with 17,561 tokens; Reverse KL is 5.55 points lower while using 6,187 more tokens.Source: §5.4 · p. 11, §5.5 · pp. 11–12

5.6 Does the Privileged Teacher Need to Co-Evolve?

  • Co-evolving teachers outperform frozen teachers with and without SRCL at both 8B and 4B, while frozen runs regress after brief initial gains.DCE reaches 65.76% and 60.00% with co-evolution versus 47.99% and 25.28% when frozen; DCE+SRCL reaches 65.97% and 61.88% versus 41.04% and 27.29%.Source: §5.6 · p. 12, Figure 5 · p. 12

5.7 Which Representation of the Verified Solution Works Better?

  • Assistant-side privileged-solution prefill improves Average by 9.79, 6.39, and 14.66 points at 8B, 4B, and 1.7B, respectively, versus user-side placement.At 8B it is also 780 tokens shorter; placement alone is insufficient, since reference-last performs better under a frozen teacher.Source: §5.7 · p. 12, Table 5 · p. 12

5.8 How Does the Method Perform under Tight Reasoning Budgets?

  • Under matched 8K and 16K budgets, Cue improves every tested DCE setting while slightly reducing mean output, whereas forcing OPSD to use more tokens is ineffective.At 8K, DCE+SRCL rises from 28.54% to 35.07%; at 16K it rises from 52.01% to 57.64%.Source: Table 6 · p. 13, §5.8 · p. 13

5.9 How Sensitive Is Performance to Decoding Hyperparameters?

  • Among tested decoding choices, temperature 1.0 gives the best Average by default, while repetition penalty 1.08 reaches a locally higher 67.15% with fewer tokens.The repetition-penalty result gains 1.18 points with 388 fewer tokens; default decoding is retained for the main comparisons.Source: §5.9 · p. 13

6 Conclusion

DCE makes privileged self-distillation recursive, and SRCL adds shorter verified rewrites; together they support recursive improvement across model scales. Qwen3-14B also shows strong early gains, with output length increasing at a later checkpoint.

6 Conclusion

  • DCE makes each updated checkpoint the next student and detached teacher, allowing revision learned in one round to shape the next round’s supervision.SRCL complements DCE with shorter, answer-verified rewrites of the model’s own responses.Source: §6 · p. 13
  • Table 7 evaluates Qwen3-8B accuracy and mean tokens while varying temperature at ρ = 1.0 and repetition penalty at T = 1.0.The caption identifies the default and row-best settings but provides no values in this passage.Source: Table 7 · p. 14

A Does Recursive Improvement Scale to a Larger Model?

  • 66.39% Average accuracy at step 30 is 45.00 percentage points above Qwen3-14B initialization; step 50 retains 65.21% as mean output grows to 20,050 tokens.The improvement spans all four benchmarks, with AIME26 rising 50.28 percentage points, from 19.44% to 69.72%.Source: A Does Recursive Improvement Scale to… · p. 17, Table 8 · p. 17

B Gemma-4-12B-IT Training Details and Additional Results

The additional results compare privileged-context placement and teacher-refresh schedules, alongside Qwen3 trajectory and SRCL-alone results. Assistant-side conditioning and fully dynamic updates lead their reported comparisons, while SRCL without DCE yields very low accuracy.

B Gemma-4-12B-IT Training Details and Additional Results

  • Matched Gemma-4-12B-IT DCE and DCE+SRCL runs differ at the objective level in λS, while holding the listed training setup and λG constant.The runs use the same 14,717-problem OPSD shard, assistant-side privileged conditioning, and 200-step LoRA training with seed 42.Source: B Gemma-4-12B-IT Training Details and… · p. 17, Table 10 · p. 18
  • 63.61% Average@12 with assistant-side conditioning exceeds 55.76% for user-side reference-last and 54.31% for user-side instruction-last on Gemma-4-12B-IT.The assistant-side run continues improving through step 100, while both user-side variants reach their best observed result at step 10.Source: B Gemma-4-12B-IT Training Details and… · p. 17, Table 9 · p. 18

C Complete Qwen3-8B and Qwen3-4B Trajectories

D Teacher-Update Schedules

  • 65.97% Average@12 with a fully dynamic teacher and 17,561 tokens performs best overall among the reported Qwen3-8B refresh strategies.EMA d = 0.3 reaches 65.35%; periodic snapshots reach at most 63.47%, and the passage cautions that each schedule has one training trajectory.Source: Table 13 · p. 19, D Teacher-Update Schedules · p. 20, Figure 6 · p. 20

E Effect of Privileged-Context Placement

The placement results favor dynamic over frozen teachers and assistant-side privileged conditioning. Position-resolved probes distinguish intervention strength from support for a fixed correct continuation, with interpretation bounded by the probe design.

E Effect of Privileged-Context Placement

  • Every dynamic placement variant outperforms its frozen counterpart; with a dynamic teacher, assistant-side prefill is strongest, reference-last intermediate, and instruction-last weakest.Under a frozen teacher, reference-last performs best but remains far below the dynamic variants.Source: E Effect of Privileged-Context Placement · p. 20, Table 16 · p. 21
  • At positions 1–32, assistant-side conditioning has the highest KL/NLL; after position 32, both metrics are lowest, indicating agreement with the fixed continuation rather than correctness.The passage describes the pattern as consistent with route initialization and notes that placement alone is insufficient under a frozen teacher.Source: E Effect of Privileged-Context Placement · p. 21, Table 17 · p. 21
  • Assistant-side conditioning’s KL rises from 0.1303 to 0.4371 and ΔNLL from +0.0414 to +0.6474; its ΔNLL is largest at every checkpoint.KL measures intervention magnitude, while ΔNLL measures whether privileged conditioning increases likelihood on the fixed correct continuation.Source: E Effect of Privileged-Context Placement · pp. 21–22, Table 18 · p. 21
  • The placement curves do not isolate a causal effect of prompt position because each column comes from a separately trained model.ΔNLL measures alignment with fixed correct trajectories, not final-answer accuracy or support for every valid derivation.Source: E Effect of Privileged-Context Placement · p. 22

F Qwen3-1.7B Training Stability

G Fixed-Trace Probe Values

  • The fixed-trace probe covers incorrect-response cohorts from AIME24, AIME25, and AIME26, including 583 observed revision events on AIME26.Figure 4 summarizes endpoint EOS and reflection-cue probabilities; table 20 reports AIME26 values.Source: G Fixed-Trace Probe Values · p. 22, Table 20 · p. 23

H EOS-Penalty Ablation

The EOS-Penalty ablation applies direct termination pressure as an alternative to SRCL. It shortens outputs, but the reported accuracy costs exceed those for concise verified rewrites.

H EOS-Penalty Ablation

  • The ablation truncates a verified response after its final balanced boxed expression and retains it only if the answer judge accepts it.The loss is averaged over eligible responses and set to zero when none are available; it is evaluated as an alternative to SRCL.Source: H EOS-Penalty Ablation · p. 23
  • Tables 21 and 22 report task-level EOS-Penalty accuracy and mean generated tokens, including checkpoint-wise Average@12 accuracy and output length.The captions specify the measures but provide no numerical values in these passages.Source: Table 21 · p. 23, Table 22 · p. 23
  • At 8B, EOS Penalty is 2,979 tokens shorter than DCE but 4.65 points less accurate, and it remains below DCE+SRCL.At 4B, EOS Penalty reaches 32.71% with 7,169 tokens.Source: H EOS-Penalty Ablation · p. 23, Table 21 · p. 23

I Cross-Scale Evaluation of OPSD-TTS

At exact 8K and 16K output budgets, OPSD-TTS gains little from doubling the forced budget and remains far below DCE.

I Cross-Scale Evaluation of OPSD-TTS

  • Only 0.07, 1.05, and 2.29 Average@12 points are gained by doubling the forced output budget at 1.7B, 4B, and 8B, respectively, and results remain far below DCE.The comparison indicates that additional generation alone does not reproduce the benefit of co-evolving training.Source: I Cross-Scale Evaluation of OPSD-TTS · p. 24, Table 23 · p. 24

J Reproducibility Details

The main experiments specify shared training and evaluation settings, while the 4B sensitivity sweep identifies a balance between guidance and concise-target learning.

J Reproducibility Details

  • Experiments use Qwen3 models from 1.7B to 14B, 14,717 OpenThoughts problems, and a privileged branch conditioned on reference tokens and the retained on-policy prefix.Gradients are stopped through the privileged branch; the main guidance objective is Forward KL.Source: J Reproducibility Details · p. 24
  • Main Qwen3 DCE optimization uses AdamW, bfloat16, learning rate 5 × 10−6, batch size 16, gradient clipping, and rank-128 LoRA across eight H100 GPUs.The table summarizes settings used throughout the main experiments, with ablation deviations stated separately.Source: J Reproducibility Details · pp. 24–25, Table 24 · p. 24
  • Evaluation uses four 30-problem benchmarks with 12 samples per problem, non-thinking decoding, and a 32K generation cap; accuracy uses verified final answers.The protocol produces 1,440 generations total, and token counts exclude prompts.Source: J Reproducibility Details · p. 25

K Additional SRCL-Weight Sensitivity

L Results on Additional Benchmarks

DCE+SRCL improves over OPSD on three additional benchmarks at both tested scales; the accompanying rewrite audit and prompt templates describe how concise targets and privileged contexts are formed.

L Results on Additional Benchmarks

  • DCE+SRCL improves over OPSD on all three additional benchmarks at both scales, with the largest gains on AMC23, while accuracy generally accompanies longer outputs.The broader-task output-length pattern contrasts with concision gains on the main benchmark suite.Source: L Results on Additional Benchmarks · p. 25, Table 26 · p. 25

M SRCL Rewrite Filtering and Acceptance

  • 2,296 of 3,200 Qwen3-8B rewrite candidates are retained (71.75%); accepted targets are 83.74% shorter than source rollouts averaging 1,997 tokens.Acceptance stays between 70.63% and 73.13% across four consecutive 50-step windows; rejected rewrites receive no SRCL loss, while their examples still receive DCE training.Source: M SRCL Rewrite Filtering and Acceptance · p. 26, Table 27 · p. 26
  • SRCL retains rewrites only when they are shorter, naturally terminated, self-contained, and structurally valid with a correct final boxed answer.Filtering also rejects explicit revision language, continuation fragments, and excessive repetition; endpoint verification compares the extracted answer with the gold answer.Source: M SRCL Rewrite Filtering and Acceptance · p. 26

N Prompt Templates

  • Privileged prompt variants use the same problem, verified solution, transition text, and on-policy response, changing only message role and ordering; SRCL uses a separate rewrite request.The placement ablation compares assistant-side context with user-side placements before or after the task instruction.Source: N Prompt Templates · p. 26, Figure 7 · p. 27

Related papers

Loading 2609.30652v1…