PaperlayerView exact revision on arXiv

Source-linked AI summary

SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning

Kenan Tang, Andong Hua, Chengxuan Qian, Saket Tiwari, Yao Qin

arXiv:2610.11132v1cs.CLcs.AIcs.LG

TL;DR

SFT can cause models to lose general capabilities their parent models already possess, particularly limiting queries that require both specialized and general abilities. SFT-as-context gives the parent model an SFT model’s response as context and prompts it to generate the final answer, without further training. SFT-as-context achieves an average recovery rate of 94.2% for general capabilities across instruction-following, multilingual tasks, question answering, and planning.

  • Problem

    SFT can cause models to lose general capabilities their parent models already possess, particularly limiting queries that require both specialized and general abilities.

    Source: §1 · p. 1

  • Method

    SFT-as-context gives the parent model an SFT model’s response as context and prompts it to generate the final answer, without further training.

    Source: §1 · p. 2

  • Results

    SFT-as-context achieves an average recovery rate of 94.2% for general capabilities across instruction-following, multilingual tasks, question answering, and planning.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Text input-output access lets responses from a small open-source SFT model improve a strong closed-source model, including when it is not the SFT model’s parent.

    Source: §1 · p. 3

    The Bayesian analysis assumes SFT is on a single domain and that the parent assigns that domain a low posterior probability for in-domain queries.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

Supervised fine-tuning (SFT) equips large language models (LLMs) with specialized capabilities, but often comes at the cost of forgetting the general capabilities of their parent models (i.e., the pretrained models before fine-tuning). This trade-off is especially limiting for queries that require both specialized and general capabilities. We introduce SFT-as-context, a training-free method in which the parent model uses the SFT model's response as context to answer the query. This allows the parent model to acquire fine-tuned capabilities from the SFT response through in-context learning while preserving its own general capabilities. Across 19 parent-SFT model pairs and 11 benchmarks, SFT-as-context remains close to the SFT models on fine-tuned capabilities, with gaps of only 2.2 and 2.1 percentage points on AIME 2024 and LiveCodeBench and 2.0 macro MAE on NutriBench-English, while staying within 2.2 percentage points of the parent models on general capabilities on average. Remarkably, it can solve queries requiring both fine-tuned and general capabilities, even when neither the parent nor SFT model succeeds alone. This approach also extends beyond parent-SFT pairs: responses from a small open-source SFT model can improve a strong closed-source LLM, outperforming either model alone. Furthermore, we use a Bayesian framework to derive theoretical guarantees that bound the error of SFT-as-context relative to the SFT model on fine-tuned capabilities and to the parent model on general capabilities. In addition, we visualize the attention weights and find that the parent model attends more to useful SFT responses and less to irrelevant ones, suggesting that selective attention helps the parent model use the SFT response through in-context learning.

1 INTRODUCTION

SFT can improve specialized performance while eroding general capabilities, a trade-off that is especially problematic when one query requires both. SFT-as-context uses an SFT model’s response to help its parent answer, and the paper reports recovery of fine-tuning gains alongside strong general-capability performance.

1 INTRODUCTION

  • SFT’s specialized gains can come with lost general abilities, leaving neither the SFT nor parent model able to meet a query requiring both.For nutrition queries, the SFT model fails to respond in the user’s language, while the parent lacks the improved nutrition accuracy; SFT-as-context satisfies both requirements.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • SFT-as-context provides a fine-tuned model’s response to its parent as context, then prompts the parent to generate the final answer.The method is training-free and uses in-context learning.Source: §1 · p. 2
  • 95.0% and 94.1% recovery rates on AIME 2024 and LiveCodeBench, respectively, accompany a 94.2% average recovery rate for general capabilities.On NutriBench-Non-English, recovery rates are 84.6% for nutrition estimation and 95.2% for language consistency.Source: §1 · p. 3
  • Macro MAE on NutriBench-English falls from 17.44 to 14.08 when Gemini 3.5 Flash uses responses from fine-tuned Qwen3-4B as context.This outperforms either model alone and does not require the SFT model to be the closed-source model’s parent.Source: §1 · p. 3
  • The parent assigns 37.44% attention to SFT responses for nutrition queries versus 15.11% for unrelated queries, consistent with selective use of the responses.The paper interprets this pattern as the parent selectively acquiring fine-tuned capabilities through in-context learning while preserving general capabilities.Source: §1 · p. 3

2 SFT MODELS FORGET GENERAL CAPABILITIES

The evaluation covers mathematical reasoning, coding, and nutrition estimation, examining fine-tuned capabilities, general capabilities, and queries that require both. The reported results show that SFT improves specialized performance while substantially degrading general capabilities.

2.1 EXPERIMENT SETUP

  • The study evaluates 19 parent-SFT pairs across mathematical reasoning, coding, and nutrition estimation, distinguishing capabilities improved by SFT from those already present in the parent.The pairs include eight math SFT models, eight coding SFT models, and three nutrition SFT models.Source: §2.1 · p. 3
  • Fine-tuned capabilities are tested on held-out data, while additional tasks combine specialization with general requirements such as formatting or answering in the query’s language.The joint-capability settings include MathIF, LiveCodeBenchIF, and NutriBench-Non-English.Source: §2.1 · p. 3

2.2 SFT CAUSES FORGETTING

  • 33.2 percentage points is the average reduction in general-capability scores after SFT; Qwen3-8B’s non-food-query score falls from 99.6% to 10.7%.The paper describes the latter as catastrophic forgetting of the parent’s instruction-following capability.Source: §2.2 · p. 4
  • On MathIF, SFT raises math correctness from 40.6% to 65.0% but lowers Hard IF from 50.1% to 21.3% by failing formatting constraints.Similar problems are reported on LiveCodeBenchIF and NutriBench-Non-English.Source: §2.2 · p. 4
  • Figure 2 separates fine-tuned from general capabilities and compares SFT-versus-parent and SFT-as-context-versus-parent performance across model pairs.Its heatmaps show larger improvements in darker blue and more severe forgetting in darker red.Source: Figure 2 · p. 5

3 SFT-AS-CONTEXT SUCCESSFULLY MITIGATES FORGETTING

SFT-as-context uses an SFT response to help the parent model generate a final answer, aiming to retain fine-tuned gains while mitigating forgetting. Across evaluated model pairs and benchmarks, it stays close to both models and can combine fine-tuned and general capabilities in one response.

3 SFT-AS-CONTEXT SUCCESSFULLY MITIGATES FORGETTING

  • SFT-as-context runs the SFT model first, then gives its response and the original query to the parent model to generate the final answer.The parent prompt also includes instructions to treat the SFT response as helpful unless it contains concrete errors.Source: §3 · pp. 5–6
  • The recovery rate measures how much of the performance gap between parent and SFT models SFT-as-context recovers, with 0% matching the worse model and 100% the better model.It can exceed 100% if SFT-as-context outperforms both; for lower-is-better metrics such as MAE, the rate is defined as 100% − R.Source: §3 · p. 6
  • Across 19 parent-fine-tuned model pairs, SFT-as-context preserves 92.0% of fine-tuning gains on average and recovers 94.2% of the general-capability performance gap.It can also combine fine-tuned and general capabilities in one response, which neither model achieves alone.Source: §3 · p. 6
  • A 0.8-percentage-point CoQA gap from the parent accompanies retention of 39.8 out of 41.8 percentage points of LiveCodeBench gains for OCR-Nemotron-1.1-14B.SFT had reduced CoQA performance by 73.4 percentage points relative to the parent.Source: §3 · p. 6
  • On MathIF, the 32B parent exhibits the boxed-answer formatting failure less often than the 7B parent, with average Hard IF gaps of −1.8 versus −7.5.The 7B parent often returns only a boxed answer, violating requests for specific words or sections.Source: §3 · p. 6

4 UNDERSTANDING THE MECHANISMS OF SFT-AS-CONTEXT

The paper analyzes SFT-as-context with a Bayesian account of domain inference and an attention-weight study. The theoretical results bound its error relative to the SFT model on in-domain queries and the parent model on out-of-domain queries; measured attention is higher for useful than hallucinated SFT responses.

4.1 THEORETICAL EXPLANATION

  • The analysis models response generation as predicting a domain from the query and then generating conditioned on that domain; conditioning on the SFT response changes the domain posterior while preserving per-domain output laws.The parent is modeled as having capabilities across many domains, while the SFT model is fine-tuned on one domain.Source: §4.1 · p. 7
  • Under the stated assumptions, the in-domain guarantees show lower error for SFT-as-context than the parent and close error to the SFT model.The assumptions include a low parent posterior for the SFT domain, a response that raises that posterior, separation from non-SFT parent outputs, and SFT expertise in its domain.Source: §4.1 · pp. 7–8
  • For out-of-domain queries, the error difference between SFT-as-context and the parent is upper-bounded by −log(1 − ρo), under Assumption 5.The paper interprets this bound as SFT-as-context performing similarly to the parent on out-of-domain queries.Source: §4.1 · pp. 7–8

4.2 ATTENTION ANALYSIS

  • Across 100 sampled queries in each category, the parent model allocated 22.33 percentage points more average attention to useful SFT responses than to potentially hallucinated nutrition estimates.The study averaged attention to SFT responses over response tokens, layers, and attention heads.Source: §4.2 · p. 8
  • The attention difference suggests the parent can use useful SFT information through in-context learning while limiting the influence of irrelevant context.The paper presents selective attention as a possible explanation for mitigating forgetting while preserving fine-tuned capabilities.Source: §4 · p. 6, §4.2 · p. 8

5 DISCUSSION

The discussion extends SFT-as-context to proprietary models and tests whether its gains arise merely from its instructions or an extra inference pass. Open-source SFT responses improve a proprietary model on NutriBench-English, while self-context controls fail to reproduce the method's balance of capabilities.

5 DISCUSSION

  • The discussion identifies proprietary-model extension and controls against instruction-only or extra-pass explanations as the two additional analyses.The method requires only text input-output access, without model weights or the fine-tuning process.Source: §5 · p. 8, §5.1 · p. 9

5.1 SFT-AS-CONTEXT FURTHER IMPROVES PROPRIETARY MODELS

  • Using Qwen3-4B context gives the best macro MAE on NutriBench-English in Table 2, and open-source SFT context consistently improves the proprietary model over either model alone.The proprietary model can combine task-specific information from the SFT response with its own capabilities rather than merely copying that response.Source: Table 2 · p. 9, §5.1 · p. 9

5.2 ABLATION STUDIES

  • Parent self-context yields English and non-English macro MAE of 36.6 and 77.4, versus 18.4 and 63.6 for SFT-as-context.The parent self-context preserves strong general capabilities but does not achieve low nutrition-estimation MAE.Source: §5.2 · p. 9
  • SFT self-context reaches only 1.3% language consistency and 17.9% non-food acknowledgment, versus 73.7% and 98.0% for SFT-as-context.The controls test whether the gains come simply from the instructions and an additional inference pass; results suggest they do not.Source: §5.2 · p. 9

6 RELATED WORK

Related work addresses forgetting through changes to training or through inference-time model combination. SFT-as-context differs by requiring only text input-output access and by focusing on SFT-induced forgetting.

6 RELATED WORK

  • Prior forgetting-mitigation approaches modify training through regularization, replay, parameter-update constraints, or adaptation-algorithm choice.The paper focuses on SFT-induced forgetting, which it describes as typically more severe than forgetting after on-policy reinforcement learning.Source: §6 · p. 9
  • Emulated Fine-Tuning (EFT) and Proxy-Tuning combine pretrained and fine-tuned predictions through token-level probabilities or logits, requiring output distributions during decoding.The paper says these approaches do not specifically target forgetting.Source: §6 · p. 10
  • Concurrent work CPR trains a router to select between parent and fine-tuned models during decoding, whereas SFT-as-context uses the fine-tuned response as parent-model context through text input-output access.The paper also notes that its inference-time approach could in principle use a response from a reinforcement-learning-fine-tuned model.Source: §6 · pp. 9–10

7 CONCLUSION

SFT-as-context uses a fine-tuned model’s response to help its parent model retain specialized capabilities while mitigating forgetting of general capabilities. The appendix details the evaluation resources, model pairs, nutrition fine-tuning, inference settings, and reproducibility provisions.

7 CONCLUSION

  • SFT-as-context is training-free and uses Bayesian and attention analyses to examine how useful SFT responses steer parent models while limiting influence on unrelated queries.The authors identify extension to other post-training methods, such as reinforcement learning, as future work.Source: §7 · p. 10

A EXPERIMENT DETAILS

  • Most results use public checkpoints and benchmarks; the exceptions are trained nutrition models, added multilingual NutriBench queries, and manually written instructions, with reproduction details provided.The paper points to Appendix K for theoretical results and proofs.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 BENCHMARKS

  • Evaluation includes conversational question answering with 500 CoQA validation questions and ACPBench’s seven reasoning tasks across 13 domains.The appendix also specifies math, instruction-following, multilingual, coding, and nutrition benchmarks, including constructed instruction-following variants.Source: §A.1 · p. 17

A.2 LIST OF SFT MODELS

  • The math and coding experiments use instruction-tuned Qwen2.5 and Qwen2.5-Coder parents, while the paper uses “pretrained model” as shorthand for an instruction-tuned starting model.This shorthand differs from the stricter usage of “pretrained” for a base model before instruction tuning.Source: §A.2 · pp. 17–18, Table 4 · p. 18

A.3 NUTRITION ESTIMATION MODEL TRAINING

  • Nutrition SFT trains Gemma-4-E4B-it, Qwen3-4B, and Qwen3-8B with full-parameter updates on 40,000 English meal descriptions and 1,000 development examples.Training uses 10 epochs, or 1,570 optimizer steps.Source: §A.3 · p. 18

A.4 INFERENCE SETTINGS

  • Inference uses a 32,768-token total sequence limit, an 8,192-token response cap for NutriBench, and common decoding parameters across models.Non-greedy benchmarks are averaged over three seeds, with a different SFT-generated context in each SFT-as-context run.Source: §A.4 · pp. 18–19

A.5 METRICS

The appendix specifies distinct evaluation measures for instruction-following, judged reading comprehension, multilingual answer-prefix use, and nutrition estimation.

A.5 METRICS

  • IFEval uses prompt-level strict accuracy, while MathIF uses Hard IF under their official evaluations.Source: §A.5 · p. 19
  • For SQuAD2.0, FQuAD2.0, and CoQA, an LLM judge evaluates responses using the prompt recommended by Hua et al. (2025), with Gemini 3.5 Flash as judge.Source: §A.5 · p. 19
  • MGSM evaluation checks whether the response uses the word for “answer” in the correct non-English language.Source: §A.5 · p. 19
  • NutriBench reports macro MAE averaged over carbohydrates, fat, protein, and energy, and heuristically checks phrases for language consistency and non-food-query acknowledgment.Source: §A.5 · p. 19

B PROMPTS

The appendix describes task prompts and SFT-as-context prompts, which place the SFT response alongside the original query for the parent model. It also specifies nutrition prompt requirements, response handling, and fallback conditions.

B.1 TASK PROMPTS

  • Most benchmarks use their official prompt templates; LiveCodeBenchIF adds a requirement for exactly three comment lines.Source: §B.1 · p. 19
  • The NutriBench template asks for language consistency and explicit acknowledgment of non-food queries, but SFT models fail to preserve those general capabilities under the specified prompting.The prompt directs non-English responses to remain in the query language and unrelated inputs to be identified without inventing a meal.Source: §B.1 · p. 19

B.2 SFT-AS-CONTEXT PROMPTS

  • For SFT-as-context, the task prompt and a version of the SFT response with its thinking block stripped are inserted into the prompt template.Source: §B.2 · p. 20
  • For math and coding, the original task is repeated and the SFT response is presented as a reference; the parent should preserve it unless a concrete error or explicit requirement violation calls for minimal correction.The authors justify repeating the task prompt with an ablation study in Appendix J.Source: §B.2 · pp. 20–21
  • The nutrition prompt treats the SFT estimate as the default and says to override a value only for a specific hard error grounded in the original description.Source: §B.2 · p. 20
  • The SFT response is omitted if generation fails, repeats pathologically, or leaves no budget for the parent; the method then falls back to the parent answering the original query alone.Source: §B.2 · p. 21
  • MathIF’s fallback rate is 3.73% across eight model pairs and three seeds, far below the reported capability gaps of over 20 percentage points.The authors conclude the improvements cannot be attributed to these fallback conditions.Source: §B.2 · p. 21

C MORE EXAMPLES OF FORGETTING AND MITIGATION

C.1 MATHIF AND LIVECODEBENCHIF GENERAL FAILURES

Examples show SFT-as-context producing responses that combine fine-tuned correctness with general instruction-following when the parent and SFT models each miss one requirement. The appendix also presents cases where the method outputs only a boxed answer and therefore violates other task instructions.

C.1 MATHIF AND LIVECODEBENCHIF GENERAL FAILURES

  • In the MathIF and LiveCodeBenchIF examples, SFT-as-context gives the correct answer and formatting, whereas the parent misses the fine-tuned task and the SFT model ignores formatting.Source: §C.1 · p. 22
  • For MathIF, the parent’s remainder-theorem approach yields −8x + 3 after incorrect evaluations, while the SFT model gets the answer right but violates lowercase formatting; SFT-as-context preserves the correct derivation and lowercase notation.Source: §C.1 · p. 22
  • For LiveCodeBenchIF, the parent’s descending sort gives 29 instead of the correct 30, while the SFT model ignores the three-comment requirement; SFT-as-context retains the SFT code structure and adds exactly three comment lines.Source: §C.1 · p. 24

C.2 SFT-AS-CONTEXT FAILURE MODE: BOXED ANSWER ONLY

  • In two further examples, Qwen2.5-7B-Instruct outputs only a boxed answer, violating requirements to include the word “number” or highlight at least two sections.Source: §C.2 · pp. 25–26

D ATTENTION VISUALIZATION

Across the attention, RL, safety, routing, and token-manipulation analyses, the paper examines when context helps and where it does not. The attention analysis finds greater prompt-directed attention to useful than hallucinated context, while the other studies identify limits under safety conflicts and token-level baselines.

D ATTENTION VISUALIZATION

  • The attention analysis normalizes each response token’s attention over prompt tokens, then averages the shares across response tokens, heads, and layers.Chat wrappers and special tokens are excluded.Source: D ATTENTION VISUALIZATION · pp. 27–28

E RL MODELS FORGET LESS

  • For the tested DeepSeek-R1-Distill-Qwen pairs, RL post-training induces negligible IFEval instruction-following forgetting, so the authors do not apply context-based methods to close the small gaps.The comparison uses prompt-level strict accuracy on IFEval.Source: E RL MODELS FORGET LESS · p. 28, Table 5 · p. 28

D ATTENTION VISUALIZATION

  • Useful context receives a higher mean share of prompt-directed attention than hallucinated context: 29.95% versus 15.61%.For non-food queries, the parent attends more to instruction tokens such as “unrelated” and “irrelevant.”Source: D ATTENTION VISUALIZATION · p. 28, Figure 3 · p. 29

F A CASE STUDY ON SAFETY

  • When the context is adversarial, SFT-as-context largely preserves harmful responses and recovers only 16.44% of the parent’s safety capability on average.The authors suggest that the instructions do not explicitly impose safety requirements, so the parent may check task compliance without first assessing safety.Source: F A CASE STUDY ON SAFETY · pp. 29–30, Table 6 · p. 30

G FORGETTING CANNOT BE MITIGATED BY ORACLE ROUTING

H TOKEN-LEVEL MANIPULATION FAILS TO MITIGATE FORGETTING FOR NUTRIBENCH

  • On NutriBench, ensembling and contrastive decoding sometimes outperform SFT-as-context on fine-tuned capabilities but neither recovers general capabilities.The authors link failures to the SFT models’ tendency to assign high probability to “contributes,” causing language inconsistency or hallucinated nutrition values.Source: H TOKEN-LEVEL MANIPULATION FAILS TO… · p. 31

I SFT-AS-CONTEXT INCURS MODEST OVERHEAD COMPARED TO SFT

The paper reports modest generation overhead and an instruction-following benefit from repeating task instructions, alongside a Bayesian account that frames SFT-as-context as evidence for domain inference. It defines model errors using KL divergence from an optimal output distribution.

I SFT-AS-CONTEXT INCURS MODEST OVERHEAD COMPARED TO SFT

  • For math and coding pairs, the second-pass response-token ratio is at most 13.0% of the SFT response-token count.Nutrition estimation is an exception: the parent generates a longer second response while comparing the SFT reference with its own estimate, though total token counts remain comparable to other tasks.Source: I SFT-AS-CONTEXT INCURS MODEST… · p. 31, Table 10 · p. 33

J ADDITIONAL ABLATION STUDIES

  • Repeating the original task instructions significantly helps instruction-following on MathIF while barely affecting fine-tuned math capability.The ablation removes the repeated instructions from the SFT-as-context prompt.Source: J ADDITIONAL ABLATION STUDIES · p. 32, Table 11 · p. 33

K THEORETICAL RESULTS AND PROOFS

  • The theoretical analysis models outputs through a latent domain variable, with fine-tuning aimed at improving the capability associated with its training domain.The parent model may represent multiple domain capabilities.Source: K THEORETICAL RESULTS AND PROOFS · p. 32, §K.1 · p. 32

K.1 ADDITIONAL INTUITION FOR THE BAYESIAN FRAMEWORK

  • In the Bayesian interpretation, the SFT response and instructions provide evidence for domain inference, changing the domain posterior while leaving conditional response distributions fixed.The paper presents this as a probabilistic interpretation, not a claim that the model explicitly performs those steps.Source: §K.1 · p. 33

K.2 DEFINING THE ERRORS

  • The theoretical analysis defines parent, SFT, and SFT-as-context errors as KL divergences from the optimal output distribution to each model’s output distribution.The SFT-as-context distribution is conditioned on the query, instructions, and SFT response.Source: §K.2 · p. 34

K.3 SFT-AS-CONTEXT IN-DOMAIN ERROR

The theoretical results give conditional error guarantees for in-domain and out-of-domain queries. In-domain, SFT-as-context can improve on the parent and remain close to the SFT model; out-of-domain, its additional error is bounded when posterior retention holds.

K.3 SFT-AS-CONTEXT IN-DOMAIN ERROR

  • Under the stated in-domain assumptions, Theorem 3 bounds SFT-as-context error below the parent-model error when its stated condition is satisfied.The assumptions include domain-posterior bounds and output separability.Source: §K.3 · p. 35

K.4 IN-DOMAIN CLOSENESS OF SFT-AS-CONTEXT TO SFT

  • Under realizability and posterior-concentration assumptions, Theorem 4 bounds the in-domain error difference between SFT-as-context and SFT without requiring output separability.The passage identifies this as a closeness guarantee to the SFT model.Source: §K.4 · p. 36

K.5 SFT-AS-CONTEXT OUT-OF-DOMAIN ERROR

  • For out-of-domain queries, the additional SFT-as-context error is small when the context only weakly suppresses the parent’s domain posterior.The guarantee requires an out-of-domain posterior-retention assumption; membership of the optimal domain in the parent’s capabilities alone does not imply it.Source: §K.5 · pp. 36–37

Related papers

Loading 2610.11132v1…