PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving Without Stopping: On-Policy Distillation at Small Scale

Hongyang Li, Yiming Zhu, Xiao Li, Caesar Wu, Said Mammar, Pascal Bouvry

arXiv:2609.37326v1cs.AIcs.CL

TL;DR

On-policy distillation is widely used to transfer reasoning to small models, but what small students take from a thinking teacher is not well understood. The paper tracks solving and stopping while distilling Qwen3-8B into three student sizes across thinking and non-thinking modes. Solving improves within two ceilings, while thinking-mode distillation filters existing stops rather than teaching new ones.

  • Problem

    What on-policy distillation transfers from a thinking teacher to small students is not well understood, and end-point scores alone cannot distinguish inability to solve from stopping too rarely.

    Source: §1 · p. 1

  • Method

    The study distills Qwen3-8B into Qwen3 4B, 1.7B and 0.6B students in thinking and non-thinking modes, tracking answering, correctness and stopping.

    Source: §1 · pp. 1–2

  • Results

    Distilled p@1 stays below the student’s starting p@n in all 27 groups, while in thinking mode students stop closing reasoning early and the teacher signals stops almost only where students already stop.

    Source: §3 · p. 3, §4 · pp. 5–6

  • Takeaways & Limitations

    At small scale, distillation transfers solving but not reliably stopping in thinking mode; the smallest students often reach a correct value without committing to it.

    Source: §7 · p. 9

Abstract

from arXiv · show

On-policy distillation, where a student learns from a stronger teacher's feedback on its own outputs, is a common way to pass reasoning to smaller models. We analyze what it transfers at small scale, distilling Qwen3-8B into Qwen3 4B, 1.7B and 0.6B students, in thinking mode (reason at length, then end the reasoning and answer) and, for comparison, in non-thinking mode (no separate reasoning phase). Long reasoning needs two abilities, solving a problem and knowing when it is solved, and we find that distillation transfers the first, but in thinking mode not the second. Solving improves at every size, up to two ceilings, which we measure comprehensively across both modes and all student sizes: a student's single attempt never exceeds what it could already reach in many attempts before training, and the smaller the student, the further it stays below the teacher. Stopping is where the modes part. In non-thinking mode every student keeps stopping; in thinking mode students stop ending their reasoning early in training, and the smaller the student, the less of this ability survives: the teacher signals a stop almost only where a student already ends its reasoning, so distillation teaches no new stops; it only keeps the student's existing stops that land on a right answer, and a weak student has few such stops. The smallest students often reach the right value but do not commit to it: they either rarely mark it or mark it and write past it. Together, these results describe how small students behave under on-policy distillation, and a diagnostic that separates answer marking, correctness and stopping.

1 INTRODUCTION

The paper asks whether on-policy distillation passes both problem-solving and knowing when to stop to small students, and follows both abilities rather than relying on end-point scores alone. Across modes and student sizes, it finds that solving transfers while stopping does not reliably transfer in thinking mode.

1 INTRODUCTION

  • The study separates solving a problem from recognizing when it is solved, because end-point scores alone cannot distinguish poor solving from stopping too rarely.It tracks whether students answer, whether answers are correct, and whether they stop.Source: §1 · pp. 1–2
  • Distilling Qwen3-8B into 4B, 1.7B and 0.6B students tests what small models learn from a stronger model’s feedback on their own outputs.The comparison includes thinking and non-thinking modes, with two sources of fine-tuning data for thinking-mode students.Source: §1 · p. 1
  • Distillation raises students’ single-attempt accuracy toward, but never beyond, their own pre-training many-attempt reach, while their share of teacher accuracy falls with size.The gain comes mainly from problems students could already solve sometimes; the set they can solve at all barely grows.Source: §1 · p. 2
  • In thinking mode, students stop closing their reasoning early in training; the teacher reinforces stops where students already stop, leaving the smallest students with almost none.The smallest students often reach the right value without committing to it: they rarely mark it from the external-SFT start, or mark it and write past it from the teacher-SFT start.Source: §1 · p. 2

2 SETUP

The setup compares three Qwen3 student sizes distilled from Qwen3-8B across thinking and non-thinking routes. It defines the response-level measures used to separate answer marking, correctness and stopping.

2 SETUP

  • Qwen3-8B serves as teacher for 4B, 1.7B and 0.6B students; thinking-mode students receive supervised fine-tuning before distillation, while pretrained-only students use the non-thinking route.Thinking-mode starts use either short external solutions or the teacher’s correct solutions.Source: §2 · p. 3
  • For each prompt, the student generates one response, the teacher scores its tokens, and a clipped one-sample reverse-KL estimate supplies the advantage for a PPO-style update without task reward.The stop token is scored like any other token, so stopping pressure depends on the teacher’s probability of stopping at the student’s current point.Source: §2 · p. 3
  • Evaluation uses lenient scoring: a sample counts as correct if any marked answer is correct, while closure rate and truncation are reported separately from scores.A marked answer must be presented as an answer, not merely appear as an intermediate value in the working.Source: §2 · p. 3

3 THE CEILING OF ON-POLICY DISTILLATION

Distillation improves students’ single-attempt accuracy, but it does not exceed their pre-training many-attempt reach, and smaller students remain further below the teacher. The gains largely come from problems already within reach, while thinking-mode students’ reduced answer marking limits improvement at smaller sizes.

3 THE CEILING OF ON-POLICY DISTILLATION

  • In all 27 route-size-benchmark groups, distilled p@1 stays below the starting student’s p@n, reaching 0.50 to 0.87 of that starting reach across benchmarks.This is the first ceiling: one distilled attempt does not exceed what the student could already solve in many attempts before training.Source: §3 · p. 3, Figure 2 · p. 4
  • Distilled p@1 reaches 0.922, 0.739 and 0.551 of the non-thinking teacher’s from 4B to 0.6B, and 0.760, 0.458 and 0.343 of the thinking teacher’s from the external-SFT start.On AIME 2026, the 1.7B and 0.6B thinking-mode students reach a tenth of the teacher’s p@1 or less; on GSM8K-200 their p@n comes close to the teacher’s at every size.Source: §3 · p. 3, Figure 3 · p. 4
  • Three-benchmark p@n changes by about 0.1 or less for non-thinking and external-SFT starts, with clearer increases for 4B on AIME 2026 and pretrained-only 0.6B on GSM8K-200.The 4B thinking-mode AIME 2026 p@n rises from 0.400 to 0.700, and to 0.733 from the teacher-SFT start; pretrained-only 0.6B GSM8K-200 rises from 0.745 to 0.945.Source: §3 · pp. 3–4, Figure 4 · p. 5
  • The gain comes mainly from problems already solved at least once before training; previously unreachable problems account for at most an eighth of the p@1 gain.At 1.7B and 0.6B from the external-SFT start, more problems leave the reachable set than enter it: 11 against 3, and 22 against 9.Source: §3 · pp. 4–5
  • In thinking mode, answers are more often right at every size, but the 1.7B and 0.6B mark answers less often, limiting their p@1 gains to +0.034 and +0.056.The 4B marks answers about as often as before (−0.006), and its p@1 rises by +0.119; missing answers for the smaller students come from samples that never stopped.Source: §3 · p. 5

4 WHY SMALL STUDENTS DO NOT LEARN TO STOP

Thinking-mode students lose closure early, while the teacher signals a stop largely where students already stop; distillation preserves some correct stops rather than creating new ones.

WHY SMALL STUDENTS DO NOT LEARN TO STOP

  • Across benchmark problems, gains mostly come from problems the starting student solved sometimes; problems it never solved mostly remain unsolved.Small students from the external-SFT start barely improve on the middle band of problems they sometimes solved.Source: Figure 5 · p. 6
  • Closure falls sharply at step 6 in every external-SFT thinking run, from 0.617 to 0.211 at 4B, 0.445 to 0.128 at 1.7B, and 0.583 to 0.113 at 0.6B.The responses reach the rollout budget in the same step; none of the tested thinking-mode variations prevents the collapse.Source: §4 · p. 5, Figure 6 · p. 6
  • The closure collapse is not confined to one setup: the 0.6B collapses at 7,168- and 16,384-token budgets, while teacher-SFT students fill the budget from the first step.Non-thinking students keep stopping throughout training.Source: §4 · p. 6, Figure 6 · p. 6
  • At student closure points, the teacher’s </think> probability is 0.976 at 4B, 0.764 at 1.7B, and 0.673 at 0.6B, but along budget-bound responses its maximum is at most about 10^-7.Thus the teacher reinforces existing closures but provides almost no stop signal along responses that do not close.Source: §4 · p. 6, Figure 7 · p. 7
  • For the officially post-trained 0.6B, lengthening alone predicts correct stops falling from 0.305 to 0.229, whereas distillation leaves them at 0.311 and removes mostly wrong stops.Across external-SFT starts, stop reliability ranges from 0.521 at 4B to 0.042 at 0.6B, and late closure is lowest for the smallest student.Source: Figure 8 · p. 7, §4 · p. 7

5 WHAT THE SAMPLES THAT DO NOT STOP CONTAIN

Unfinished samples often contain a correct value without marking it, while students from the teacher-SFT start can mark answers but write past them.

WHAT THE SAMPLES THAT DO NOT STOP CONTAIN

  • At 1.7B and 0.6B, samples with no marked answer rise from 0.373 to 0.518 and from 0.328 to 0.579, while marked right answers rise only a little.At 0.6B, samples whose marked answers are all wrong shrink from 0.433 to 0.127; the increase in unmarked samples runs to the evaluation limit.Source: §5 · p. 8
  • Among unfinished external-SFT samples, the correct value appears in 0.211, 0.298, and 0.240 at 4B, 1.7B, and 0.6B—four to seven times chance—and often appears early.The students then continue for tens of thousands of further characters; the 4B’s unfinished training responses contain the correct value in 0.309 versus 0.019 by chance.Source: §5 · p. 8
  • Forcing a stop lets the officially post-trained 0.6B answer correctly on 0.364 of problems its starting model solved three or four times in four, but only 0.049 of problems it never solved.From the teacher-SFT start, the 1.7B and 0.6B never finish evaluation samples, mark 185.5 and 150.3 answers per sample, and sometimes overwrite correct answers with wrong ones.Source: §5 · p. 8

6 RELATED WORK

Prior work studies on-policy distillation variants and stopping as separate concerns; this paper examines what distillation teaches about stopping and how far students can rise. Related work also frames stopping as a decision that small students may not reach in the teacher’s states.

RELATED WORK

7 CONCLUSION

Across three student sizes, two modes, and two fine-tuning data sources, on-policy distillation transfers solving but not thinking-mode stopping. The results separate student reach from teacher reach and show that the smallest students often reach a correct value without committing to it.

CONCLUSION

  • Across three sizes, both modes, and two fine-tuning sources, distillation improves solving but does not teach stopping in thinking mode.Solving rises up to the student’s pretraining reach and a share of teacher performance that shrinks with student size.Source: §7 · p. 9
  • Non-thinking students keep stopping, while less stopping survives in smaller thinking-mode students, whose unfinished outputs may contain the right value without a commitment.The paper’s diagnostic separates answering, correctness, and stopping; an end-point score alone would read lost stops as lost solving.Source: §7 · p. 9
  • The paper locates the stopping problem along responses that run on, where the teacher gives no signal to stop, rather than at the stop token.The authors frame this as guidance for where methods for small students should act.Source: §7 · p. 9

AI USE STATEMENT

The authors used generative AI mainly to improve the paper’s writing and help write analysis and plotting scripts, while retaining responsibility for the study and its claims.

AI USE STATEMENT

  • Generative AI helped improve the paper’s writing and write analysis and plotting scripts.Source: AI USE STATEMENT · p. 9
  • The authors designed the study, ran all training and evaluation, and checked every analysis, number, and claim against logs and data.Source: AI USE STATEMENT · p. 9
  • Generative AI was not used to generate data or run experiments.Source: AI USE STATEMENT · p. 9

ETHICS STATEMENT

The study evaluates mathematical reasoning between openly released models using public benchmarks and training sources, without human subjects, personal data, or new data collection.

ETHICS STATEMENT

  • The study distills mathematical reasoning between openly released language models and evaluates it on public benchmarks.Source: ETHICS STATEMENT · p. 10
  • It involves no human subjects, personal data, or new data collection.Source: ETHICS STATEMENT · p. 10
  • Training prompts come from a public competition-mathematics set, while cold-start solutions come from a public corpus or the teacher.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper documents its models, training data, scoring rules, and evaluation statistics. Its experimental pipeline uses cleaned public math problems and two thinking-format cold starts.

REPRODUCIBILITY STATEMENT

  • The teacher and pretrained-only and officially post-trained student checkpoints are public; code, derived data, and run configurations are slated for release upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXPERIMENTAL DETAILS

  • Cleaning and deduplication reduced 17,398 DAPO-Math-17k problems to 13,597 training problems, excluding overlaps with evaluation sets.Source: A EXPERIMENTAL DETAILS · p. 13
  • Thinking-format cold starts use either 5,878 short external solutions or 1,524 correct teacher solutions, with different length limits and mean lengths.External solutions average 1,530 tokens and are at most 2,048 tokens; teacher solutions average 4,809 tokens and are at most 7,000 tokens.Source: A EXPERIMENTAL DETAILS · p. 13

B SCORING AND STATISTICS

  • Marked answers are matched to references after LaTeX normalization and numeric-value comparison; lenient, strict, and committed scores distinguish answer correctness and completion.Samples without marked answers count as wrong under all three scores.Source: B SCORING AND STATISTICS · p. 13
  • Lenient scoring can reward multiple different marked answers, but that credit is usually at most 0.025 of the three-benchmark mean.The exception is the 1.7B and 0.6B teacher-SFT-start students, which mark over a hundred answers per sample.Source: B SCORING AND STATISTICS · p. 14
  • The correct-value diagnostic checks for a qualifying reference value anywhere in unfinished samples and is not part of any score.It restricts analysis to reference integers of at least three digits absent from the prompt and reports the frequency of another problem’s answer as a comparison.Source: B SCORING AND STATISTICS · p. 15
  • GPQA-Diamond is evaluated only on the non-thinking route using the committed answer, with strict scoring reported beside it and no averaging with other benchmarks.The leading option letter of each marked answer is scored.Source: B SCORING AND STATISTICS · p. 15
  • Paired 95% intervals use 10,000 problem-level bootstrap resamples, with shared draws for comparisons and unweighted averaging across benchmarks.Differences whose intervals contain zero are treated as no difference; changes of about 0.1 or less are called small even when significant.Source: B SCORING AND STATISTICS · p. 15

C SUPPLEMENTARY RESULTS: HOW FAR SOLVING RISES

Across benchmarks, distillation raises students’ single-attempt accuracy but does not exceed their starting many-sample solve rates. The supplementary results also show that gains and answer-marking changes vary across modes, sizes, and benchmarks.

C SUPPLEMENTARY RESULTS: HOW FAR SOLVING RISES

  • Distillation raises every student’s mean lenient p@1, but no benchmark’s distilled p@1 exceeds that student’s starting p@n.Table 5 reports benchmark-level lenient p@1 and p@n; p@n uses 48 samples on AIME 2026 and AMC23 and 32 on GSM8K-200.Source: Table 5 · p. 15, C SUPPLEMENTARY RESULTS: HOW FAR… · p. 15
  • On AMC23 and GSM8K-200 the starting and distilled pass@k curves close as k grows; on AIME 2026 the distilled curve leads for the 4B thinking model but trails at large k for the external-SFT-start 0.6B.One distilled attempt matches the starting model’s best of 4 attempts at 4B, and 2 at 1.7B and 0.6B from the external-SFT start.Source: C SUPPLEMENTARY RESULTS: HOW FAR… · pp. 15–16, Figure 10 · p. 16, Table 6 · p. 16
  • No distilled student’s p@1 exceeds its starting p@n on any benchmark; the four interval-reaching-zero cases are all on AIME 2026 and have scores close to zero.The starting p@n changes by 0.2 or more only for the 4B thinking model on AIME 2026 and pretrained-only 0.6B on GSM8K-200.Source: C SUPPLEMENTARY RESULTS: HOW FAR… · p. 17, Table 8 · p. 17
  • Answer accuracy rises on every route and size, while marked-answer rates rise on the non-thinking route and teacher-SFT starts but fall for external-SFT thinking students at 1.7B and 0.6B.Source: C SUPPLEMENTARY RESULTS: HOW FAR… · p. 17, Table 9 · p. 18
  • Majority-vote accuracy rises from 0.656 to 0.792 for the 4B external-SFT start, while the 0.6B vote is unchanged at 0.400 to 0.398.At 1.7B, thinking routes tie the non-thinking route under majority vote despite lower single-sample performance.Source: C SUPPLEMENTARY RESULTS: HOW FAR… · p. 17
  • On non-thinking GPQA-Diamond, distillation raises all three sizes, but only the 4B student ends above the 0.250 chance level.The table scores the last marked answer and reports paired 95% bootstrap intervals.Source: C SUPPLEMENTARY RESULTS: HOW FAR… · p. 17, Table 10 · p. 18

D SUPPLEMENTARY RESULTS: WHY SMALL STUDENTS DO NOT LEARN TO STOP

The supplement traces early losses of thinking-mode closure, probes the teacher’s stop signal, and examines answers in samples that do not stop. It also reports data and evaluation caveats relevant to interpreting the results.

D SUPPLEMENTARY RESULTS: WHY SMALL STUDENTS DO NOT LEARN TO STOP

  • External-SFT thinking runs lose closure at step 6 with a 7,168-token budget as responses fill the budget and marked answers disappear; the officially post-trained 0.6B partly recovers.Runs at 16,384 tokens drop at step 7; runs from the teacher-SFT start and the 1,024-token run begin with little closure.Source: D SUPPLEMENTARY RESULTS: WHY SMALL… · p. 18, Figure 11 · p. 19
  • The officially post-trained 0.6B keeps more than half its closure over 20 steps despite stop reliability of about one third; 4B still ends during evaluation at 30,720 tokens.The 4B's evaluation marked-answer rate does not fall because its longer reasoning still ends within that limit.Source: D SUPPLEMENTARY RESULTS: WHY SMALL… · p. 19

D.1 PROBES

  • On identical prefixes, 0.6B ranks </think> first only where it already closed; lengthening alone removes most wrong closes but also a quarter of correct ones.The lengthening null model does not reproduce the distilled model's preservation of correct closes.Source: §D.1 · pp. 19–20, Table 14 · p. 20, Table 15 · p. 20
  • The teacher endorses students’ existing closures but gives almost no stop signal along trajectories truncated at the budget.The probe scores the teacher's probability of </think> at student closure positions and its maximum probability along responses that reach the budget.Source: §D.1 · p. 19, Table 13 · p. 20

E SUPPLEMENTARY RESULTS: WHAT THE SAMPLES THAT DO NOT STOP

F DATA NOTES

  • 856 of 13,597 training problems (6.3%) have statements cut short; GSM8K was not decontaminated, and AMC23 served for validation and evaluation.All runs except two easy-to-hard curriculum runs train on the full set; evaluated checkpoints were taken at fixed steps, and paired runs used the same data order.Source: F DATA NOTES · p. 22
Loading 2609.37326v1…