PaperlayerView exact revision on arXiv

Source-linked AI summary

Continually Post-training Small Models via Synthetic Self-distillation

Woosung Koh, Hyunsoo Lee, Kyungjae Lee, Haeju Park, Dahyun Lee, Dasol Hwang, Moontae Lee

arXiv:2610.13113v1cs.LGcs.AI

TL;DR

Continual post-training must add capabilities while retaining existing ones, but SDFT’s dependence on in-context learning and instruction-following limits it at smaller scales. SynthSelf outsources an in-distribution rewrite to a minimally larger, same-family aligner, and it outperforms on- and off-policy distillation in acquisition while preserving prior capabilities across benchmarks.

  • Problem

    Continual post-training must add capabilities while retaining existing ones, and SDFT’s reliance on in-context learning and instruction-following is a limitation for smaller models.

    Source: §1 · p. 1

  • Method

    SynthSelf outsources an in-distribution rewrite to a minimally larger, same-family aligner and uses off-policy supervision with rejection sampling.

    Source: §1 · p. 2, §2.4 · p. 4

  • Results

    Across four settings and three base scales, SynthSelf outperforms on- and off-policy distillation in acquisition by +5.7%p to +16.9%p, while preserving prior capabilities on 14 benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Rejection sampling can shrink the training set while improving acquisition over SFT, indicating gains from supervision quality rather than demonstration quantity.

    Source: §4 · p. 9

    SynthSelf requires a larger, same-family aligner; adapting out-of-family aligners remains a direction for future work.

    Source: §4 · p. 9

Abstract

from arXiv · show

Continual post-training faces a dual challenge of acquisition and retention: models must learn new capabilities while preserving prior knowledge. While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with in-context demonstrations, it relies on strong in-context learning and instruction-following abilities. Consequently, SDFT fails on smaller models critical for low-latency, on-device use. To enable continual learning at smaller scales, we first identify the root cause of failure. We show that poor acquisition and forgetting correlate linearly with the post-trained model's distribution shift, revealing the necessity of in-distribution learning. We thus introduce Synthetic Self-Distillation (SynthSelf), imitating SDFT's in-distribution rewrite by outsourcing the task to a minimally larger, same-family aligner. Coupled with off-policy supervision and rejection sampling, this provides the high-quality supervision small models require without breaking their original distribution. Evaluations spanning 4 acquisition datasets (including a mixture) and 14 retention benchmarks show SynthSelf consistently outperforms 6 methods (including off- and on-policy distillation) across 3 small model scales. Extensive further analyses confirm SynthSelf paves the way toward effective small-model continual learning.

1 INTRODUCTION

The introduction argues that distribution shift is tied to both acquisition and forgetting, then presents SynthSelf as an in-distribution rewrite outsourced to a larger, same-family aligner. Across small model scales, it reports broad acquisition gains while retaining prior capabilities.

  • SDFT leverages in-context learning and instruction-following for self-supervision, but remains infeasible at smaller scales where those abilities are weak.Source: §1 · p. 1
  • Across three base sizes, SynthSelf achieves best acquisition across science, tool-use, medicine, and a four-subject science mixture while preserving capabilities on 14 retention benchmarks against four baselines.Source: Figure 1 · p. 2, §1 · p. 2
  • Post-training distribution shift linearly correlates with poor acquisition and forgetting; transforming fixed data toward in-distribution forms monotonically improves math retention, math performance, and coding retention.Source: §1 · p. 2, Figure 2 · p. 3
  • SynthSelf imitates self-distillation’s in-distribution rewrite by outsourcing it to a larger model, using off-policy supervision to enable rejection sampling and higher-quality supervision for small models.Source: §1 · p. 2
  • +5.7%p to +16.9%p: SynthSelf outperforms both on- and off-policy distillation in acquisition, averaged over four settings and three base scales when teacher size is controlled.Source: §1 · p. 2
  • A smaller, weaker in-distribution aligner outperforms a larger, stronger out-of-distribution teacher while reducing alignment FLOPs by 69.9%.Source: §1 · p. 2
  • Against prompt-matched SDFT, SynthSelf gains +18.96%p in acquisition and +1.68%p in retention, showing the reported improvement is not due to prompt differences.Source: §1 · p. 2
  • Ablations confirm that combining rejection sampling with off-policy supervision is a meaningful component of the method.Source: §1 · p. 2

2 METHOD

Continual fine-tuning seeks to add capabilities while preserving existing ones. SynthSelf uses a larger, same-family aligner to create in-distribution rewrites, then applies off-policy training with rejection sampling.

2.1 PROBLEM SETTING: CONTINUAL FINE-TUNING

  • Continual fine-tuning aims to improve performance on a same-domain test set while retaining performance on existing capabilities.The setup starts from a post-trained policy and trains it further on new knowledge.Source: §2.1 · p. 3

2.2.1 IN-DISTRIBUTION LEARNING IS KEY FOR CONTINUAL LEARNING

  • Across 23 math datasets, forgetting strongly correlates with policy distribution shift after fine-tuning (Spearman ρ = −0.85).The datasets were grouped by domain and release date, and forgetting was measured across five non-math benchmarks.Source: §2.2.1 · pp. 3–4
  • With the corpus and prompts held fixed, higher-shift targets track with in-domain and catastrophic forgetting, while low-shift self-generated solutions improve math acquisition and preserve coding capabilities.The three target variants differ in reasoning length and source: full R1 traces, final solutions, and verified-correct model-generated solutions.Source: §2.2.1 · p. 4

2.3 LIMITATIONS OF SELF-DISTILLATION

  • SDFT’s performance relative to SFT decreases monotonically with student size, and smaller models produce weaker self-supervision.Across prompts, average accuracy falls to 60%, and chain-of-thought traces are dropped 8.2% of the time.Source: §2.3 · p. 4, Figure 3 · p. 4

2.4 OFF-POLICY IN-DISTRIBUTION DISTILLATION AS SYNTHETIC SELF-DISTILLATION

  • SynthSelf generates aligner rewrites and uses off-policy supervision to enable rejection sampling, filtering for accurate examples before fine-tuning the base model.The approach targets both in-distribution alignment and high-quality supervision; the aligner rewrites each example, and accepted samples form the offline training set.Source: §2.4 · pp. 4, 6
  • Same-family aligners provide more in-distribution rewrites than Gold SFT, while stronger cross-family models fail to meaningfully improve alignment with the base model.Same-family Qwen2.5 aligners have the lowest average KL, below the Gold SFT baseline, and higher data accuracy than the base model.Source: Figure 4 · p. 5, §2.4 · p. 6
  • Off-policy rejection sampling is a valuable component, and SynthSelf outperforms standard off-policy distillation even when model scale is controlled.The ablation caption reports these comparisons without specifying numerical values.Source: Figure 5

3 EMPIRICAL STUDY

The empirical study evaluates SYNTHSELF across four acquisition settings and 14 retention tasks, then analyzes aligner choice, prompt matching, and rejection sampling. SYNTHSELF leads acquisition while matching the strongest baselines on retention, and further analyses examine the role of in-distribution supervision.

3.1 EXPERIMENTAL PROTOCOL

  • The evaluation covers science, tool-use, medicine, and a four-subject science mixture, using Qwen2.5-Instruct base models and the SDFT baselines plus SFT, DFT, and ASFT.The protocol also compares against off- and on-policy distillation and fixes minimally tuned SFT hyperparameters across methods.Source: §3 · p. 6, §3.1 · pp. 6–8

3.2 RESULTS

  • SYNTHSELF leads acquisition in all 4 benchmarks and ties the best baseline on 14 retention tasks across 4 baselines, distillation comparisons, and three base models.The four acquisition benchmarks include a science mixture; the retention tasks span 14 benchmarks.Source: Table 1 · p. 7, §3.1 · p. 7

3.3 FURTHER EXPERIMENTAL ANALYSES

  • More in-distribution aligners, measured by NLL, predict performance better than aligner data-generation accuracy, while same-family aligners reduce generation inference costs by 62.9% to 76.9%.Larger out-of-distribution aligners can generate higher-quality supervision yet introduce severe distributional shift and yield similar or worse acquisition.Source: Table 2 · p. 8, §3.3 · pp. 8–9

3.2 RESULTS

  • Across all four settings, SYNTHSELF improves acquisition by +5.7%p to +16.9%p over off- and on-policy distillation while maintaining comparable retention, with both baselines using the identical aligner.Off- and on-policy distillation differ only in conditioning; the authors attribute gains primarily to in-distribution demonstration injection.Source: §3.2 · p. 8

3.3 FURTHER EXPERIMENTAL ANALYSES

  • SYNTHSELF exceeds prompt-matched SDFT by +18.96%p in acquisition and +1.68%p in retention on average, and outperforms the tested prompt variants.The comparison includes four SDFT prompt variants and reports higher acquisition across all three domains and higher average retention than every variant.Source: Figure 6 · p. 9, §3.3 · p. 9
  • Rejection-sampling filtering consistently improves both SYNTHSELF and off-policy distillation.The authors hypothesize that filtering is especially important for smaller base models, which are sensitive to noisy supervision.Source: §3.3 · p. 9

4 DISCUSSION

The discussion contrasts SYNTHSELF’s supervision source with related approaches, describes compute and performance trade-offs, and identifies its aligner requirement and future directions.

4 DISCUSSION

  • Unlike a related method combining self-rewrites, similarity filtering, cross-family fusion, and an on-policy KL regularizer, SYNTHSELF uses a same-family aligner to address degradation in student-generated rewrites.The authors also distinguish their approach from work on distribution drift that does not directly address continual learning or small language models.Source: §4 · p. 9
  • Rejection sampling can reduce training compute by shrinking the dataset, while SYNTHSELF improves acquisition by +4.2%p on average despite using fewer demonstrations than SFT.The authors attribute the gains to supervision quality rather than quantity and note that increasing pass@k trades inference compute for broader supervision coverage.Source: §4 · p. 9
  • SYNTHSELF requires a larger, same-family aligner; the authors suggest adapting out-of-family aligners to produce in-distribution supervision as future work.They note that many major open-weight model families provide a spectrum of sizes and propose including the base model’s answer in-context as one possible direction.Source: §4 · p. 9
  • The authors used generative AI for experimental code generation and grammar and spelling review, and generated reasoning demonstrations for three SciKnowEval subjects.They state that AI was not used for conceptualization, methodology design, data analysis, or result interpretation.Source: AI USE STATEMENT · p. 10

APPENDIX

The appendix lists prompt variants used for SYNTHSELF training and for self-rewrite and prompt-matched SDFT studies.

A PROMPTS

  • Prompt A is the rewrite instruction used for every trained SYNTHSELF cell, while Prompts B, C, and D are variants used in the self-rewrite study and prompt-matched SDFT ablation.The prompt placeholders {q} and {gold} stand for the question and gold response.Source: A PROMPTS · p. 18
  • Other variants request a rewritten response with its thinking process and specify a reasoning-and-answer format or require the same final answer.The appendix lists reasoning and answer tags for one output format.Source: A PROMPTS · p. 18
  • The tool-use variants ask for a rewrite that preserves the final answer, with some versions also requiring the original format or the thinking process.The listed output formats include an Action and Action Input structure.Source: A PROMPTS · pp. 18–19

B SETUP FOR THE IN-DISTRIBUTION EXPERIMENTS

The setup describes two experiments that share a fine-tuning configuration and differ only in training data.

B SETUP FOR THE IN-DISTRIBUTION EXPERIMENTS

C.1 HARDWARE

Experiments ran on A100 nodes, with commonly available CPU, RAM, and disk hardware that did not induce bottlenecks.

C.1 HARDWARE

C.2 DATASETS

Datasets followed the SDFT paper’s defaults; SCIKNOWEVAL demonstrations were model-generated, and its gold subset was split into training and test data with evaluation prompts filtered accordingly.

C.2 DATASETS

  • All datasets used default values following the SDFT paper.Source: §C.2 · p. 19
  • Because SCIKNOWEVAL supplies only answer keys for multiple-choice rows, demonstrations were model-generated: GPT-4o for CHEMISTRY and gpt-5.6-luna for the other three subjects.Source: §C.2 · p. 19
  • The SCIKNOWEVAL gold subset comprises L-3 examples with correct demonstrations, split 80/20 into training and test data.Source: §C.2 · p. 19
  • Evaluation prompts were filtered to match that split, while every arm was scored on the same pool.Source: §C.2 · p. 19

C.3 TRAINING CONFIGURATION

The configuration describes 23 Experiment 1 corpora, a SCIENCE-MIX train/test split, and fixed settings for every experimental cell.

C.3 TRAINING CONFIGURATION

  • Tab. 5 settings were fixed to a single seed, 42, chosen a priori and never changed after experimentation.Source: §C.3 · p. 19, Table 5 · p. 20
  • Experiment 1 used 23 corpora, named by their HuggingFace loading paths and grouped by domain and upload date relative to the base model’s 2024-09 cutoff.Repository creation dates were read from the HuggingFace API.Source: Table 3 · p. 20
  • The SCIENCE-MIX corpus split SCIKNOWEVAL L-3 gold examples 80/20 at seed 0.Source: Table 4 · p. 20
  • Tab. 5 lists configurations for every cell in Tabs. 1 and 2; unlisted values use library defaults.Source: Table 5 · p. 20

D QUALITATIVE EXAMPLE OF ALIGNER REWRITES

The qualitative example shows a Science question and its gold demonstration alongside the aligner rewrite used for SynthSelf training.

D QUALITATIVE EXAMPLE OF ALIGNER REWRITES

Loading 2610.13113v1…