PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillFactory: Self-Distillation For Learning Cognitive Behaviors

Zayne Sprague, Jack Lu, Manya Wadhwa, Sedrick Keh, Mengye Ren, Greg Durrett

arXiv:2512.04072v2cs.CLcs.AI

TL;DR

Existing approaches can require stronger models or other substantial resources to teach reasoning skills that a base model does not exhibit, leaving unclear how to prime those skills before RL. SkillFactory rearranges a model’s own samples into skill-demonstrating SFT traces, and its initialization supports post-RL generalization to harder variants and novel domains while improving robustness to out-of-domain regression.

  • Problem

    When base models do not exhibit reasoning skills, existing approaches can require stronger models, substantial training, or custom data, and it is unclear whether post-SFT improvements reflect better skill learning during RL.

    Source: §1 · p. 1

  • Method

    SkillFactory prompts and restructures a model’s own samples into SFT traces demonstrating verification and retries, without distilling from a larger model.

    Source: §1 · p. 1

  • Results

    SkillFactory initialization supports post-RL generalization to harder variants and novel domains, matching or exceeding strong baselines, and greater resilience to out-of-domain regression.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that self-distillation can make diverse reasoning skills more accessible without stronger-model distillation.

    Source: §8 · p. 10

Abstract

from arXiv · show

Reasoning models leveraging long chains of thought employ various cognitive skills, such as verification of their answers, backtracking, retrying by an alternate method, and more. Previous work has shown that when a base language model exhibits these skills, training that model further with reinforcement learning (RL) can learn to leverage them. How can we get models to leverage skills that aren't exhibited by base models? Our work, SkillFactory, is a method for fine-tuning models to roughly learn these skills during a supervised fine-tuning (SFT) stage prior to RL. Our approach does not rely on distillation from a stronger model, but instead uses samples from the model itself, rearranged to provide training data in the format of those skills. These "silver" SFT traces may be imperfect, but are nevertheless effective for priming a model to acquire skills during RL. Our evaluation shows that (1) starting from SkillFactory SFT initialization helps a model to generalize to harder variants of a task post-RL, despite lower performance pre-RL;(2) cognitive skills are indeed used by the model; (3) RLed SkillFactory models are more robust to regression on out-of-domain tasks than RLed base models. Our work suggests that inductive biases learned prior to RL help models learn robust cognitive skill use.

1 INTRODUCTION

Reasoning models benefit from skills such as verification and retrying, but existing routes to develop them can require stronger models, substantial training, or custom data. SkillFactory rearranges a model’s own outputs into skill-demonstrating traces for SFT before RL, with reported gains in post-RL generalization and robustness.

Motivation

  • Prior approaches can require stronger models, substantial training, custom pre-training data, or complex combinations, while the effect of post-SFT gains on learning skills during RL remains unclear.Source: §1 · p. 1

Approach

  • SkillFactory constructs “silver” traces by prompting and restructuring a model’s own samples to demonstrate verification and retries, without distilling from a larger model.Source: §1 · p. 1
  • SFT is intended to prime skill structure rather than maximize pre-RL task performance; RL then hones the skills, and higher pre-RL performance need not predict higher post-RL performance.Source: §1 · p. 2

Contributions

  • SkillFactory initialization enables post-RL generalization to harder variants and novel domains, matching or exceeding strong baselines, and greater resilience to out-of-domain regression.The authors report these contributions across Countdown and OpenThoughts training settings.Source: §1 · p. 2

2 BACKGROUND AND MOTIVATION

The paper focuses on retrying and reflection as reasoning skills, noting that these behaviors may be inconsistently expressed and that their usefulness depends on the task. It motivates explicit skill marking and evaluates how search-focused training affects transfer and out-of-domain performance.

2.1 COGNITIVE SKILLS IN LLMS

  • Retrying restarts inference after an extracted answer, whereas reflection separately evaluates that answer; together, these behaviors extend reasoning beyond a single attempt.Source: §2.1 · pp. 2–3
  • Some base models show implicit verification and restarting, but these skills surface less consistently when expressed incidentally in natural language.Source: §2.1 · p. 3
  • Explicit tags mark reflection and retrying, and reflection reruns the computation, potentially mitigating errors from the initial search.Source: §2.1 · p. 3
  • Prior approaches include sparse-reward RL, stronger-model distillation, and targeted data curation; SkillFactory instead generates data from the base model and emphasizes structure for consistent skill generalization.Source: §2.1 · p. 3

2.2 TASKS: PLANNING, SEARCH, AND COMPUTATION

  • Search-focused tasks make answers easier to check than to find, but training on them trades off transfer: light training may not transfer skills beyond similar tasks, while heavier training can degrade broader out-of-domain performance.Source: §2.2 · pp. 3–4
  • Verification and retrying may benefit multiplication and CommonsenseQA despite not appearing naturally, while those tasks can require skills other than search.Source: §2.2 · p. 4

3 SKILLFACTORY

SkillFactory curates a model’s own solution attempts and reflections into structured SFT traces that demonstrate verification and retries. It then uses these traces as a starting point for RL rather than expecting SFT itself to raise task performance.

Framework

  • SkillFactory has three stages: base-model inference with skill-specific heuristics, SFT on the resulting traces, and sparse-correctness-reward RL using off-the-shelf algorithms such as GRPO.Source: §3 · p. 4

Data curation

  • The data-curation process samples diverse solutions from the base model, generates reflections that assess them, and combines the pairs into structured traces with explicit retry and verification.Source: §3 · p. 4

Solution generation

  • For each question, four chain-of-thought prompts yield 64 solution attempts, which are parsed and checked against the ground-truth answer to form correct and incorrect solution pools.Source: §3 · pp. 4–5

Reflection generation

  • The model generates four reflections per solution, and only reflections whose verdict matches the solution’s actual correctness are retained.Source: §3 · p. 5

Trace construction

  • Trace construction shuffles correct and incorrect solution-reflection pairs, appends a correct pair last, and formats the sequence with tags and transition phrases, producing traces that eventually succeed.Source: §3 · p. 5

4 EXPERIMENTAL SETUP

The experiments compare SkillFactory with several warm-start and RL baselines across Countdown and other reasoning tasks. The primary Countdown regime uses training on Countdown-3arg and evaluates transfer to harder variants and held-out tasks.

4.1 BASELINES

  • The Countdown experiments use three base models and compare SkillFactory with RL-only training, externally curated BOLT data, stronger-model distillation, and STaR self-distillation.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 TASK SETUP AND EVALUATION

  • Evaluation covers Countdown and Letter Countdown, acronym generation, multiplication, CommonsenseQA, GSM8K, and challenging math and science datasets; held-out and harder variants test generalization.Source: §4.2 · p. 6

4.3 TRAINING SETTINGS

  • The primary regime creates SFT data from 4,000 Countdown-3arg rows and uses 1,000 held-out Countdown-3arg questions for RL, with three base models.Source: §4.3 · p. 6
  • A second regime uses 1,000 or 10,000 OpenThoughts rows for SFT and 10,000 held-out rows for RL; Table 1 averages held-out Countdown and OOD task difficulties for Qwen2.5-1.5B-Instruct.Source: §4.3 · pp. 6–7, Table 1 · p. 7

5 RESULTS

SkillFactory’s strongest reported gains are in transfer: it generalizes from Countdown-3arg to harder variants and performs competitively on unseen tasks and challenging math benchmarks. Results also show that its performance does not rise with more SFT data in the OpenThoughts setting.

5.3 SKILLFACTORY IMPROVES COMPLEX MATHEMATICAL REASONING

  • At 10k SFT examples, SkillFactory scores 40.6% overall on challenging math datasets, approaching QwQ distillation’s 42.5%.At 1k examples, SkillFactory surpasses QwQ distillation on AMC (37.5%) and Math500 (64.6%).Source: Table 2 · p. 8, §5.3 · p. 8

5.1 SKILLFACTORY ENABLES EASY-TO-HARD GENERALIZATION

  • 25.1% accuracy for SkillFactory →GRPO on harder Countdown variants (4–6 arguments) exceeds R1 Distill →GRPO’s 21.2% by +3.9 points.SkillFactory also improves on RL-only by 9.3%.Source: §5.1 · p. 8

5.2 SKILLFACTORY MAINTAINS ROBUSTNESS OUT-OF-DOMAIN

  • R1 Distill →GRPO slightly exceeds SkillFactory →GRPO on unseen-task OOD accuracy, 35.9% vs. 35.7%, though SkillFactory performs well on average.The gap from base models to RLed R1 models is substantially closed in two of three models.Source: §5.2 · p. 8

5.3 SKILLFACTORY IMPROVES COMPLEX MATHEMATICAL REASONING

  • SkillFactory’s overall score falls from 42.1% at 1k SFT examples to 40.6% at 10k; the authors suggest its core skills are learned early.They contrast this with distillation, where models learn new strategies and knowledge from the teacher.Source: §5.3 · p. 8

6 BUDGET FORCING

Budget forcing gives a model a second reasoning attempt after its initial token budget, extending generation up to 8,192 tokens. SkillFactory gains on Countdown, while the largest multiplication gain is reported for RL-only.

6 BUDGET FORCING

  • After an initial 4,096-token generation, a SkillFactory-specific <sample> trigger requests another reasoning attempt, allowing continuation to 8,192 tokens total.The initial budget matches RL training.Source: §6 · pp. 8–9
  • Figure 5 shows SkillFactory producing much longer thinking traces, with token-length distributions closer to R1 distillation across three tasks.The authors attribute its multiplication result to already performing multiple retries and verifications during standard inference.Source: Figure 5 · p. 9, §6 · p. 9
  • SkillFactory gains +5.3 points on Countdown with budget forcing (17.5→22.8), compared with +1.2 for RL-only and +4.7 for R1 distillation.On multiplication, RL-only gains +6.9 (24.6→31.4), versus SkillFactory’s +1.3 (35.2→36.6).Source: §6 · p. 9, Table 3 · p. 9

7 ANALYSIS

The analysis reports reflection on wrong answers, more reflection on harder Countdown variants, and explicit skill use on OOD tasks. It also finds lower OOD verifier accuracy when SFT traces lack internal ordering.

7 ANALYSIS

  • Incorrect-class reflection F1 values all exceed 0.8, indicating that wrong answers are correctly rejected.Table 4 reports answer attempts, reflections, and correct/incorrect verification F1 for SkillFactory and the No Sample Order ablation.Source: §7 · p. 9, Table 4 · p. 9
  • Countdown-4arg exhibits more reflection than Countdown-3arg, while Letter Countdown weaknesses often reflect model-scale uncertainty about English words.The analysis describes reflection as generalizing to other domains and scaling with task difficulty.Source: §7 · pp. 9–10
  • Removing internal ordering from SkillFactory SFT traces lowers verifier accuracy out of domain.The comparison is the No Sample Order ablation.Source: Table 4 · p. 9, §7 · p. 10
  • SkillFactory produces moderate, varied response lengths on Countdown-4arg and OOD tasks, while the RL baseline often gives short or degenerate OOD outputs.The analysis reports explicit skill use by SkillFactory on OOD tasks and both implicit and explicit skill use on Countdown variants.Source: §7 · p. 10

8 CONCLUSION

The conclusion presents SkillFactory as self-distillation that reshapes a model’s own outputs into retry and verification traces, without requiring a stronger teacher. The appendix specifies SFT and RL training configurations.

8 CONCLUSION

  • SkillFactory restructures a model’s own outputs into silver traces exhibiting retry and verification, improving performance on harder variants and across OOD tasks without stronger teachers.The conclusion also reports that the approach enables inference scaling methods such as budget forcing.Source: §8 · p. 10

A.1 HYPERPARAMETERS: SUPERVISED FINE-TUNING

  • SFT uses each base model’s own silver traces for two epochs, aiming to internalize sampling, reflecting, and retrying patterns rather than improve task performance at that stage.Training uses context length 4096, learning rate 1e-6, cosine annealing, full fine-tuning, and batch size 1.Source: §A.1 · p. 14

A.2 HYPERPARAMETERS: REINFORCEMENT LEARNING

  • RL uses GRPO with binary final-answer correctness rewards on 1,000 Countdown-3arg questions or 10,000 OpenThoughts questions.Both settings train for 150 steps without KL-divergence penalties.Source: §A.2 · p. 14

A.3 GENERATION PARAMETERS: DATASET CONSTRUCTION AND EVALUATION

The section specifies generation settings for SkillFactory samples and reflections, plus repeated-sampling procedures for evaluation.

A.3 GENERATION PARAMETERS: DATASET CONSTRUCTION AND EVALUATION

  • SkillFactory sample and reflection generation uses Qwen2.5-1.5B-Instruct with temperature 0.7, repetition penalty 1.1, top p 0.8, and top k 20.Source: §A.3 · p. 14
  • Most benchmarks are sampled four times, while GPQA, AIME, and AMC are sampled 34 times and their run accuracies averaged.Source: §A.3 · p. 14

B.1 ABLATIONS

The ablations examine trace ordering, reflections, prompt diversity, and an RL-only instruction-prompt variant. SkillFactory outperforms these alternatives on out-of-domain Countdown tasks.

B.1 ABLATIONS

  • The ablations remove sample ordering or reflection content, restrict solving to one prompt, or add an instruction prompt to RL-Only without SFT.Source: §B.1 · p. 14
  • 32.0% overall accuracy for SkillFactory on out-of-domain tasks exceeds 25.8% for RL-Only (Instruction Prompt).The instruction-prompt variant performs well on Countdown but degrades on 9 of 10 out-of-domain tasks.Source: §B.1 · p. 15
  • 24.1% for No Sample Order and 23.3% for No Reflections on out-of-domain tasks both fall below SkillFactory.The reported pattern supports the authors' conclusion that structured SFT traces are essential for cross-domain transfer.Source: §B.1 · p. 15
  • 29.0% overall accuracy for No Prompt Diversity remains below SkillFactory, particularly on computational tasks such as Multiplication.The authors suggest that varied reasoning patterns during SFT improve adaptation to new domains.Source: §B.1 · p. 15
  • The authors identify an explicit SFT stage and the quality of assembled traces as key elements of SkillFactory.Source: §B.1 · p. 15

C FULL RESULTS

The full-results section reports performance across evaluations for three model families trained on Countdown-3arg, including task difficulty breakdowns.

C.1 QWEN2.5-1.5B-INSTRUCT

  • For Qwen2.5-1.5B-Instruct trained on Countdown-3arg, Tables 6 and 7 report baseline performance across evaluations, including each difficulty level.Source: §C.1 · p. 15

C.2 QWEN2.5-7B-INSTRUCT

C.3 OLMO-3-7B-SFT-INSTRUCT

  • For Olmo-3-7B-SFT-Instruct trained on Countdown-3arg, Tables 10 and 11 report baseline performance across evaluations, including each difficulty level.Source: §C.3 · p. 16

D DATA CURATION

This appendix describes SkillFactory’s construction of self-generated SFT data, including response-and-reflection pairing, correctness judgments, prompt variation, and— for OpenThoughts—rationalization with hidden hints. It also covers filtering for usable boxed answers and building datasets at two scales.

D.1 GLUE PHRASES

  • Outcome-aware glue phrases are added after each sample-reflection sequence, prompting another attempt while reflecting whether the prior answer was correct, incorrect, or left unspecified.Correct sequences draw from correct or generic phrases; incorrect sequences draw from incorrect or generic phrases. An LLM generated the phrase set from hand-written seeds, followed by manual filtering and editing.Source: §D.1 · p. 16

D.2 PROMPT VARIANTS

  • Four prompt variants elicit planning and execution, alternative answers, or question rephrasing, alongside the original step-by-step prompt.The variants prescribe formats such as a high-level plan before solving, marking alternatives with “Alternatively,” or beginning with “Rewritten Question:”.Source: §D.2 · p. 18

D.3 REFLECTION PROMPTS

  • Reflection prompts ask models to assess a prior response without correcting it, briefly justify the assessment, and provide only a Correct or Incorrect verdict in tags.Examples judge answers by checking task constraints, such as whether a word uses permitted letters or whether arithmetic reaches the requested total.Source: §D.3 · pp. 19–21

D.4 RATIONALIZATION

  • For OpenThoughts sample generation, difficult questions receive answer hints that the model must not mention; responses containing “hint” are filtered out, and rationalization is not used for reflections.Source: §D.4 · p. 24
  • The rationalization prompt instructs the model not to mention the hint and to reason toward the hinted answer instead.The example hint supplies the answer 20 for an ellipse geometry problem.Source: §D.4 · p. 24

D.5 OPENTHOUGHTS

  • OpenThoughts data yields silver labels from QwQ responses; filtering removes rows without a boxed answer or with multiple boxed answers that vary, leaving around 700k instances.The final boxed span after the closing think tag is treated as the correct answer.Source: §D.5 · p. 24
  • For each question, the pipeline generates N = 2 samples per prompt variant, including rationalization prompts, for 16 samples per row, then creates one reflection per sample.The resulting SFT data pools are built at scales of 1k rows and 10k instances.Source: §D.5 · p. 24

E MODEL OUTPUTS

The output examples illustrate how SkillFactory and RL-only models exhibit implicit and explicit reflection, answer attempts, and retries, including examples from Countdown and long multiplication.

E MODEL OUTPUTS

  • SkillFactory responses show implicit and explicit reflections, whereas the RL-only Countdown response uses implicit retrying and verification.The paper describes this contrast for in-domain task variants.Source: E MODEL OUTPUTS · p. 24
  • In the Countdown example, the solution is (60 + 20) * 11 = 880; reflection verifies it by checking 60 + 20 = 80, then 80 * 11 = 880.Source: E MODEL OUTPUTS · p. 26
  • The two cited passages report answer values of 249480 and 243432, respectively; the passages themselves do not identify any method, task, or process associated with either value.Source: E MODEL OUTPUTS · pp. 27, 32
  • SkillFactory explicitly attempts answers and reflects on them, while the RL-only output presupposes a candidate answer and shows no explicit reflection or retrying.The SkillFactory example also retries with different strategies, including enumerating constraints and applying a step-by-step approach.Source: E MODEL OUTPUTS · p. 36

E.1 ANALYSIS OF SKILL USE

  • For the Countdown-3arg-only SkillFactory model, Table 13 reports answer attempts, reflections, and verification F1 for correct and incorrect classes across tasks.The accompanying text says the analysis covers all tasks.Source: Table 13 · p. 39, §E.1 · p. 39

F ADDITIONAL DETAILS FOR BOLT BASELINE

  • Claude Sonnet 4 was prompted on 10 training-split Countdown questions to produce traces with analysis, planning, alternatives, backtracking, reflection, verification, and format adherence.The questions were from Countdown with 3 arguments.Source: F ADDITIONAL DETAILS FOR BOLT BASELINE · p. 39
  • Supplementary figures show token-response distributions for SkillFactory compared with the RL Baseline, R1 Distillation, BOLT, and STaR.The figure captions identify these model pairs but do not provide distribution values in the supplied passages.Source: Figure 6 · p. 40, Figure 7 · p. 41, Figure 8 · p. 42

G LLM CONTRIBUTIONS

  • The authors used LLMs mainly for minor LaTeX tweaks and mild editing; they rewrote or heavily edited and rephrased any output.Source: G LLM CONTRIBUTIONS · p. 44

Related papers

Loading 2512.04072v2…