PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Distillation Enables Continual Learning

Idan Shenfeld, Mehul Damani, Jonas Hübotter, Pulkit Agrawal

arXiv:2601.19897v2cs.LG

TL;DR

Foundation models struggle to continually acquire new skills and knowledge without degrading existing capabilities. SDFT uses a demonstration-conditioned version of the model as its teacher to generate on-policy training signals, outperforming SFT across skill-learning and knowledge-acquisition tasks while reducing catastrophic forgetting.

  • Problem

    Foundation models need to acquire new skills and knowledge over time while preserving existing capabilities.

  • Method

    SDFT distills predictions from a demonstration-conditioned teacher model into the same model acting as an unconditioned student on its own trajectories.

  • Results

    Across skill-learning and knowledge-acquisition tasks, SDFT improves new-task performance while substantially reducing catastrophic forgetting; on strict knowledge-acquisition accuracy, it reaches 89% versus SFT’s 80%.

  • Takeaways & Limitations

    SDFT enables a single model to accumulate multiple skills sequentially without performance regression, supporting on-policy distillation as a practical approach to continual learning from demonstrations.

  • Takeaways & Limitations

    SDFT depends critically on the base model’s in-context learning capabilities and struggles when adaptation requires a fundamental shift in generation patterns.

Abstract

from arXiv · show

Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce forgetting, it requires explicit reward functions that are often unavailable. Learning from expert demonstrations, the primary alternative, is dominated by supervised fine-tuning (SFT), which is inherently off-policy. We introduce Self-Distillation Fine-Tuning (SDFT), a simple method that enables on-policy learning directly from demonstrations. SDFT leverages in-context learning by using a demonstration-conditioned model as its own teacher, generating on-policy training signals that preserve prior capabilities while acquiring new skills. Across skill learning and knowledge acquisition tasks, SDFT consistently outperforms SFT, achieving higher new-task accuracy while substantially reducing catastrophic forgetting. In sequential learning experiments, SDFT enables a single model to accumulate multiple skills over time without performance regression, establishing on-policy distillation as a practical path to continual learning from demonstrations.

1 INTRODUCTION

SDFT addresses continual learning from demonstrations by converting expert examples into on-policy signals through self-distillation, enabling new skill and knowledge acquisition while reducing catastrophic forgetting. It targets the limitations of off-policy SFT without requiring explicit reward functions.

  • Motivation: Foundation models generally remain parameter-static after deployment, limiting their ability to acquire skills, internalize knowledge, or improve from experience.Inference-time retrieval and prompting can alter behavior but do not update model parameters.
  • Motivation: Off-policy SFT can cause poor generalization and severe catastrophic forgetting during sequential adaptation, motivating on-policy learning from demonstrations.Existing on-policy methods primarily arise in reinforcement learning, where explicit reward functions provide feedback; demonstrations alone do not provide such rewards.
  • Related work: SDFT offers an alternative to inferring explicit rewards from demonstrations, avoiding the strong reward-structure priors that can limit inverse reinforcement learning’s practical use.The approach instead performs on-policy distillation directly from demonstrations.
  • Method: SDFT uses the same model as student and teacher, conditioning the teacher on expert demonstrations to generate on-policy learning signals for the student.This leverages the model’s in-context learning ability without requiring parameter updates during demonstration conditioning.
  • Evaluation: Across skill learning and knowledge acquisition, SDFT enables stable updates while substantially reducing catastrophic forgetting compared with supervised learning.These settings test learning new task performance and incorporating new information into the model.

2 RELATED WORK

Prior work motivates on-policy learning for avoiding compounding errors, improving generalization and transfer, and reducing catastrophic forgetting, while IRL addresses missing rewards through structural assumptions. SDFT is also related to context distillation and concurrent self-distillation work, but prior context-distillation methods typically use offline static-context supervision.

  • Off-policy versus On-policy Learning: On-policy learning avoids the compounding errors that arise when off-policy imitation policies encounter states outside the demonstrations.Ross et al. (2011) identifies this inference-time failure mode, which on-policy algorithms avoid by continually training on model-induced trajectories.
  • Off-policy versus On-policy Learning: On-policy reinforcement learning has been reported to improve out-of-distribution generalization, related-task transfer, and continual-learning retention relative to purely off-policy training.These empirical findings are attributed to Agarwal et al. (2024), Han et al. (2025), Chu et al. (2025), Li et al. (2025), Huan et al. (2025), Shenfeld et al. (2025), and Lai et al. (2025).
  • Inverse Reinforcement Learning: Inverse reinforcement learning uses demonstrations to infer an underlying reward, enabling on-policy updates without requiring direct cloning of expert actions (Xu et al., 2020).Successful IRL formulations rely on structural assumptions for reward identifiability, including soft-optimal expert behavior, distinguishable trajectories, or preference access (Wulfmeier et al., 2015; Ho & Ermon, 2016; Ziegler et al., 2019; Ouyang et al., 2022).
  • Context Distillation: Context distillation uses a model conditioned on additional information as a teacher for an otherwise identical student (Bai et al., 2022; Snell et al., 2022).Prior approaches typically perform offline distillation from static contexts such as few-shot examples or behavioral guidelines, supervising students on teacher-distributed trajectories.
  • Self-Distillation: Two concurrent papers independently proposed the Self-Distillation algorithm, which they view as complementary evidence for its potential in training language models (Zhao et al., 2026; H¨ubotter et al., 2026).

3 SELF-DISTILLATION FINE-TUNING

SDFT uses a demonstration-conditioned version of the model as its teacher and trains the student on-policy with reverse-KL distillation. The method is mathematically equivalent to on-policy reinforcement learning with an implicit reward inferred from the model’s demonstration-conditioned behavior, enabling sequential skill learning while retaining prior performance.

  • Method: SDFT uses the same model as teacher and student: a demonstration-conditioned teacher generates on-policy targets, and the student minimizes reverse KL divergence to that teacher.The teacher is formed through in-context learning, while student responses are sampled from the current student policy.
  • Teacher assumptions: The approach assumes that conditioning on an expert demonstration approximates the optimal next policy, requiring both near-optimal task reward and minimal KL deviation from the current policy.The minimal-deviation condition prevents the teacher from merely copying the demonstration and preserves behavior anchored to the base model.
  • Sequential continual learning: In sequential training on three tasks, SDFT learns each task while retaining performance on the others, whereas SFT performance drops as subsequent tasks are learned.Figure 3 linearly normalizes performance relative to base-model accuracy and the maximum accuracy across both algorithms.
  • Implicit-reward interpretation: The self-distillation objective is equivalent in expectation to policy-gradient optimization with an intrinsic reward defined by token-level probability changes between the student and its demonstration-conditioned counterpart.This gives SDFT an interpretation as an on-policy RL algorithm maximizing rewards inferred from the model’s own demonstration-aware behavior.
  • Empirical validation: On ToolAlpaca, the demonstration-conditioned teacher remains closer to the base policy than SFT, with KL divergence 0.68 nats versus 1.26 nats.The comparison uses D_KL(π∥π_0) as a proxy for distance from the policy during training, validating the teacher’s minimal-deviation property.

4 EXPERIMENTS

Across skill learning and knowledge acquisition, SDFT improves new-task performance while better retaining prior capabilities than offline baselines. It also supports sequential skill accumulation, reasoning-model adaptation without explicit traces, and gains that strengthen with model scale and on-policy learning.

  • Skill Learning: SDFT is the only evaluated approach that improves new-task performance without significant degradation of prior capabilities, whereas SFT causes substantial forgetting and alternatives recover it only partially.Figure 4 shows superior Pareto efficiency across all three skill-learning tasks; DFT and Re-invocation do not match SDFT’s retention of the base model.
  • Knowledge Acquisition: 89% strict accuracy versus SFT’s 80% shows SDFT substantially improves acquisition of newly injected knowledge and nearly closes the gap to oracle RAG.The base model cannot initially answer the questions, continual pretraining performs poorly, and SFT improves substantially but remains behind SDFT.
  • Robustness and Ablations: SDFT’s skill-learning gains persist across pass@k up to 128 and exceed both the base model and SFT, indicating improvements beyond superficial distributional sharpening.Offline distillation improves over SFT but consistently underperforms on-policy SDFT, highlighting the importance of on-policy learning rather than teacher quality alone.
  • Long-Horizon Continual Learning: SDFT enables stable sequential accumulation of all three skills, improving each newly introduced task while maintaining previously learned performance; SFT instead exhibits severe interference.Figure 3 reports that earlier skills rapidly degrade when SFT shifts to a new task, producing oscillating performance.
  • Scaling: At 7B and 14B, SDFT improves over SFT by four and seven points, respectively, while small 3B models lag because their in-context learning provides weak teacher guidance.The monotonic scaling trend links SDFT effectiveness to the model’s ability to perform in-context reasoning.
  • Reasoning Models: 43.7% accuracy versus 23.5% for SFT, from a 31.2% starting accuracy, shows that SDFT adapts reasoning models effectively without explicit intermediate reasoning traces.SFT also sharply shortens responses, indicating collapsed reasoning behavior, whereas the proposed method improves accuracy despite supervision containing only final answers.

5 DISCUSSION AND LIMITATIONS

SDFT complements rather than replaces on-policy RL: it learns from demonstrations without rewards, improves generation quality, and offers efficient token-level supervision. Its limitations include higher training cost, dependence on in-context learning, inherited linguistic artifacts, and residual forgetting.

  • Relationship to on-policy RL: SDFT applies when expert demonstrations are available without explicit rewards, whereas on-policy RL assumes rewards and optimizes expected return through exploration.
  • Relationship to on-policy RL: SDFT consistently improves pass@k across k, suggesting it can initialize subsequent RL fine-tuning with a stronger policy.The improvement indicates greater diversity and quality among high-probability generations.
  • Computational Costs: SDFT requires one on-policy generation per prompt and provides token- or logit-level supervision, unlike group-based advantage estimation that raises generation costs.Compared with SFT, SDFT costs approximately 2.5× more FLOPs and roughly 4× more wall-clock training time.
  • Learned Artifacts and Future Work: Students can inherit spurious teacher phrases such as “Based on the text...” even without the original context, and the practical workaround remains heuristic.A more principled solution remains an open problem; some degradation of prior capabilities also remains despite reduced forgetting.
  • Requirements for Model Capability: SDFT effectiveness depends on base-model in-context learning: smaller models with weak ICL fail to provide meaningful teacher signals.The method’s design also constrains the types of adaptations it can support.

A ADDITIONAL ABLATIONS · A.1 ESTIMATING THE KL GRADIENT

The ablation compares three estimators for the sequence-level KL gradient, whose practical computation involves bias, variance, and cost trade-offs. The full analytic per-token estimator is selected for main experiments because it offers the most stable optimization and best downstream performance, while single-trajectory sampling avoids costly negligible gains.

  • A.1 ESTIMATING THE KL GRADIENT: The sequence-level KL gradient is difficult to estimate because the current policy determines both the sampling distribution and the logarithm term.Different estimators therefore trade off bias, variance, and computational cost.
  • A.1 ESTIMATING THE KL GRADIENT: The token-level estimator decomposes KL into independently differentiated token terms but ignores how early tokens affect future distributions, making it biased.This estimator corresponds to a partial derivative of the sequence-level KL.
  • A.1 ESTIMATING THE KL GRADIENT: The full analytic per-token estimator marginalizes over the vocabulary at each timestep, reducing variance relative to sample-based token estimators but retaining sequence-level bias.It remains computationally attractive because it uses quantities already produced during the forward pass.
  • A.1 ESTIMATING THE KL GRADIENT: The Rao–Blackwellized estimator integrates analytically over next-token distributions while sampling prefixes, yielding unbiased KL and gradient estimates with provably lower variance than standard Monte Carlo.Its advantage comes at higher computational cost, and it did not improve results measurably in this setting.
  • A.1 ESTIMATING THE KL GRADIENT: The full analytic per-token estimator consistently provides the most stable optimization and best downstream performance despite sequence-level bias.The token-level estimator has higher variance and weaker KL control, while Rao–Blackwellization provides no measurable gains relative to its added complexity.
  • A.1 ESTIMATING THE KL GRADIENT: The main experiments use one trajectory per prompt with the analytic per-token KL estimator because additional trajectories negligibly improve results while substantially increasing compute.Although multiple samples can theoretically reduce Monte Carlo variance, increasing samples per prompt produced negligible practical improvements.

A.2 THE IMPORTANCE OF DEMONSTRATION-CONDITIONED CONTEXT · A.3 CHOICE OF TEACHER MODEL

The ablations show that effective knowledge transfer requires a teacher conditioned on the full demonstration context, while teacher-policy choice critically affects training stability. Full text-plus-answer conditioning outperforms partial contexts, and both frozen-base and self-teacher choices have important limitations.

  • A.2 THE IMPORTANCE OF DEMONSTRATION-CONDITIONED CONTEXT: The method differs from recent offline knowledge-injection distillation by conditioning the teacher on both source text and a worked answer and performing distillation on-policy.The cited prior approaches use only the raw corpus as teacher context.
  • A.2 THE IMPORTANCE OF DEMONSTRATION-CONDITIONED CONTEXT: 89% strict accuracy with full text-plus-answer conditioning outperforms 75% with text-only conditioning, establishing full demonstration context as critical for knowledge transfer.The full context also outperforms answers-only conditioning, while text-only conditioning remains weaker despite outperforming answers-only context.
  • A.2 THE IMPORTANCE OF DEMONSTRATION-CONDITIONED CONTEXT: The context ablation holds the on-policy training procedure fixed and compares article-text-only, answer-only, and full text-plus-answer teacher conditioning.Direct comparisons with offline distillation are deferred to Section 4.6.
  • A.3 CHOICE OF TEACHER MODEL: Teacher-policy choice critically affects stability, even though the framework does not require an external teacher.The ablation evaluates how the teacher is instantiated during distillation.
  • A.3 CHOICE OF TEACHER MODEL: Using the frozen base model as teacher gives stable training but consistently underperforms because it does not reflect improvements acquired during learning.Its limitation is that the teacher remains disconnected from the student’s newly learned capabilities.
  • A.3 CHOICE OF TEACHER MODEL: Using the student itself as teacher causes severe instability, as stochastic token-level fluctuations can amplify through the on-policy feedback loop and make training diverge.The passage reports that maintaining an exponential moving average (EMA) is investigated as an alternative teacher instantiation.

B TRAINING AND EVALUATION DETAILS · B.1 TRAINING DETAILS

Experiments used full-parameter fine-tuning on single NVIDIA H200 GPUs, with hyperparameter selection based on validation performance. SDFT benefited from multiple training epochs, whereas SFT generally overfit after one epoch; training used an analytic KL estimator with one on-policy rollout per example.

  • B.1 TRAINING DETAILS: The experiments fine-tuned all model parameters on a single NVIDIA H200 GPU using the Hugging Face TRL library.Each method underwent sweeps over learning rates, batch sizes, and training epochs.
  • B.1 TRAINING DETAILS: Reported test results came from the checkpoint with the best validation performance on the target task.
  • B.1 TRAINING DETAILS: An EMA teacher provided stable and effective training, whereas a frozen base model performed inferiorly because it failed to track learning progress.
  • B.1 TRAINING DETAILS: Using the current student directly as the teacher caused training instabilities.
  • B.1 TRAINING DETAILS: SDFT typically benefited from 2 epochs on Skill Learning tasks and 4 epochs on Knowledge Acquisition tasks, while SFT showed no gains beyond one epoch in most cases.Tables 3 and 4 report the corresponding search spaces and selected hyperparameters.
  • B.1 TRAINING DETAILS: SDFT constructed teacher contexts with the Section 3 prompt template and used an analytic per-token KL gradient estimator with one on-policy rollout per example.

B.2 EVALUATION DETAILS

Evaluation used decoding settings matched to each metric, reported results across three random seeds with 95% confidence intervals, and assessed prior capabilities on six standard benchmarks.

  • Sampling Strategy: Accuracy used greedy decoding (temperature = 0), while pass@k used temperature = 1.0 with nucleus sampling (top-p = 0.95).
  • Statistical Reporting: Unless stated otherwise, experiments used 3 random seeds and report mean performance with 95% confidence intervals.
  • Prior Capabilities Evaluation: Prior capabilities were evaluated on HellaSwag, TruthfulQA, MMLU, IFEval, Winogrande, and HumanEval using the Language Model Evaluation Harness.

B.3 DATASET DETAILS

The experiments use distinct datasets for science Q&A, tool use, medical reasoning, and knowledge acquisition, with task-specific splits, demonstration construction, and automated or exact-match evaluation procedures.

  • Science Q&A: Science Q&A uses the Chemistry L-3 SciKnowEval subset, split approximately 75%/5%/20% into train, validation, and test sets, with GPT-4o-generated valid demonstrations for every training example.Up to eight GPT-4o responses were sampled per prompt, retaining one matching the correct final answer; accuracy used exact matching for the multiple-choice task.
  • Tool Use: Tool-use experiments use ToolAlpaca’s original train-test split with demonstrations included, evaluating API calls by regex matching while allowing argument-order variation.
  • Medical: Medical experiments train on approximately 20,000 English HuatuoGPT-o1 questions and evaluate 1,000 randomly sampled verifiable questions using GPT-5-mini to judge correctness against reference answers.The evaluator labels responses CORRECT when they contain the key medical information, even with different phrasing or additional correct details, and INCORRECT otherwise.
  • Knowledge Acquisition: Knowledge acquisition uses Wikipedia articles about nine 2025 natural disasters occurring after the model’s knowledge cutoff, with GPT-5-generated multi-fact question-answer pairs and manual duplicate-question checks.The listed events include earthquakes, floods, storms, a tornado, and a hurricane; GPT-5-mini served as the automated evaluator.
Loading 2601.19897v2…