PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levels

Junjie Ye, Yuming Yang, Yang Nan, Shuo Li, Qi Zhang, Tao Gui, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan

arXiv:2509.16596v2cs.CLcs.AI

TL;DR

The effects of supervised fine-tuning on model knowledge remain insufficiently understood, limiting prediction and control of knowledge changes. The paper evaluates five LLaMA-2 and LLaMA-3 models on CBQA and analyzes fine-tuning at token and parameter levels. Data category and scale affect CBQA performance, and selectively restoring SFT updates can improve performance while preserving prior knowledge.

  • Problem

    The impact of SFT on model knowledge, including effects of data category and scale and ways to mitigate undesirable changes, remains insufficiently understood.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates five LLaMA-2 and LLaMA-3 models on CBQA across data mastery levels and scales, then analyzes token logits and selectively restores changed parameters.

    Source: §1 · p. 2

  • Results

    Across five LLMs, data category and scale unexpectedly shaped CBQA performance, while up to 90% of SFT updates were unnecessary or detrimental to knowledge enhancement.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Selectively restoring SFT updates can improve performance while preserving prior knowledge.

    Source: §6 · p. 9

    The study is phenomenological rather than a proposed efficient fine-tuning strategy, and its main analysis is limited to LLaMA-2 and LLaMA-3.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) acquire substantial world knowledge during pre-training, which is further shaped by post-training techniques such as supervised fine-tuning (SFT). However, the impact of SFT on a model's knowledge remains underexplored, limiting our ability to control knowledge change behavior in fine-tuned models. To address this gap, we evaluate closed-book question answering (CBQA) performance across five LLMs from the LLaMA-2 and LLaMA-3 families. Surprisingly, models fine-tuned on 1,920 samples perform up to 14% worse than those fine-tuned on only 240 samples. Furthermore, varying the level of knowledge mastery in the fine-tuning data leads to performance fluctuations of over 12%. To investigate these effects, we analyze model behavior at both the token and parameter levels. Our analysis reveals that up to 90% of parameter updates during SFT do not contribute to knowledge enhancement. Restoring these updates can improve performance on the CBQA task, depending on the characteristics of the fine-tuning data. These insights offer practical guidance for developing fine-tuning strategies that more effectively strengthen model knowledge.

1 Introduction

The paper asks how SFT data category and scale alter model knowledge, a gap that limits prediction and control of knowledge changes. It investigates this through CBQA evaluation and token- and parameter-level analyses.

1 Introduction

  • The impact of SFT on model knowledge remains insufficiently understood, including how data category and scale change knowledge and how undesirable effects can be mitigated.This gap limits the ability to predict and control knowledge changes in fine-tuned models.Source: §1 · pp. 1–2
  • Up to 14% worse CBQA performance followed fine-tuning with 1,920 rather than 240 samples, while data-category choice produced fluctuations over 12%.The evaluation covered five LLaMA-2 and LLaMA-3 models.Source: §1 · p. 2
  • As fine-tuning data grows, token-logit KL divergence initially decreases, then sharply rises beyond a threshold, especially for poorly mastered data, correlating with performance degradation.The token-level analysis compares fine-tuned and pre-trained model logits.Source: §1 · p. 2
  • The study evaluates five LLMs from the LLaMA-2 and LLaMA-3 families on CBQA, examining five knowledge-mastery categories and different fine-tuning data scales.It also analyzes model behavior at token and parameter levels.Source: §1 · p. 2
  • Up to 90% of SFT parameter updates do not contribute to knowledge enhancement, and restoring highly changed parameters can improve performance.Restoring up to 90% of updates did not harm performance and produced improvements exceeding 10% in some cases.Source: §1 · p. 2

2 Related Work

Prior work examines CBQA knowledge retention and hallucination, and studies SFT dataset quality and size. The paper identifies limited attention to fine-tuning’s internal dynamics and models’ prior mastery of the data.

CBQA and Model Knowledge

  • Prior CBQA studies examine retention of pre-existing knowledge, hallucinations associated with overfitting or unfamiliar data, and dataset size and quality.The cited examples include Ren et al. (2024), Gekhman et al. (2024), and Ye et al. (2024c).Source: §2 · p. 2
  • Unlike work focused mainly on dataset characteristics, this paper analyzes token- and parameter-level dynamics and identifies unnecessary updates as a factor contributing to CBQA degradation.The related-work discussion frames unnecessary parameter updates as a key factor in performance degradation on CBQA.Source: §2 · p. 2

Data Quality and Scale of SFT

  • Prior dataset work reports that small, high-quality SFT datasets can match larger ones, while scaling data quantity with quality maintained can yield further gains.The passage cites Zhou et al. (2023) and Yang et al. (2025b) on small high-quality datasets, and Kaplan et al. (2020), Chung et al. (2022), Wei et al. (2022), and Dong et al. (2024) on scaling.Source: §2 · pp. 2–3
  • Prior studies have paid limited attention to how a model’s prior knowledge of fine-tuning data and data quantity affect its knowledge.This work examines mastery levels and data scale in SFT on CBQA.Source: §2 · p. 3

3 Experiments

The experiments test how SFT data scale and knowledge mastery affect closed-book question answering across five LLaMA-2 and LLaMA-3 models. Performance peaks at 240 training samples, while training-data mastery also shapes results.

3.1 Dataset

  • Training uses 10 location-related topics; the same topics form the in-domain test set, while the other 14 topics form the out-of-domain set.The training examples are categorized by the pretrained model’s knowledge mastery level.Source: §3.1 · p. 3

3.2 Models

  • The experiments cover five decoder-only models: LLaMA-2-7B, 13B, and 70B, plus LLaMA-3-8B and 70B.The authors limit their analysis to decoder-only architectures.Source: §3.2 · p. 3

3.3 Experimental Setup

  • Mastery categorization uses 21 topic-specific templates, sampled 10 times at temperature 0.7; final accuracy is averaged across test sets and five training-data runs.Training uses batch size 8 for one epoch with AdamW and a learning rate of 1 × 10−5.Source: §3.3 · pp. 3–4

3.4 Main Results

  • CBQA performance peaks at 240 samples across training-data types, and adding more data can reduce performance.The section reports that full-category datasets produce performance near the lowest levels across categories.Source: Figure 2 · p. 4, §3.4 · pp. 4–5
  • Low-mastery training impairs high-mastery test performance, while high-mastery training underperforms on low-mastery tests; mid-level DM_train−2 balances overall results.The table covers LLaMA-3 models evaluated on in-domain and out-of-domain sets using 1,920 examples across mastery levels.Source: Table 2 · p. 5, §3.4 · p. 5

4 Token-Level Analysis

The token-level analysis compares fine-tuned and pretrained token distributions using KL divergence, then relates distribution shifts to CBQA performance. The reported patterns vary with data scale and mastery level.

4.1 KL Divergence Computation

  • KL divergence quantifies token-probability differences between fine-tuned and pretrained models, using re-normalized distributions for the first predicted token.The method selects the fine-tuned model’s top ten logits and matches their corresponding pretrained logits before applying softmax and computing divergence.Source: §4.1 · p. 6
  • For cross-model comparisons, the analysis focuses on DM test−4, where pretrained completion success exceeds 75% across samples and templates.For each topic, the authors select the mapping template with the highest success rate and examine answer tokens near the beginning of completions.Source: §4.1 · p. 6

4.2 Results Analysis

  • Greater divergence between fine-tuned and pretrained logits accompanies declining performance, with degradation on DM test−4 tracking divergence.The authors suggest that substantial shifts can mean loss of knowledge already mastered by the pretrained model, particularly with large datasets.Source: Figure 5 · p. 6, Figure 4 · p. 6, §4.2 · p. 7
  • Across data categories, logits divergence first decreases and then increases as training data grows, with low-mastery data showing stronger shifts.The text attributes the large shift at 60 samples to unstable training and the smaller discrepancy at 240 samples to improved stability.Source: §4.2 · pp. 6–7

5 Parameter-Level Analysis

The parameter-level analysis restores the most changed parameters to their pretrained values and evaluates the resulting performance. Restoration often helps, but its effects depend on fine-tuning data scale and mastery level.

5.1 Parameter Restoration

  • The method ranks parameters by their change from pretrained values, then progressively restores the largest changes while monitoring performance.Updates are concentrated: more than 70% occur in fewer than 1% of parameters.Source: §5.1 · p. 7, Table 3 · p. 7

5 Parameter-Level Analysis

  • The analysis finds that most SFT parameter updates are unnecessary and that larger datasets or lower-mastery data make models more adversely affected by unnecessary changes.The authors describe the parameter-level study as testing whether excessive updates degrade performance.Source: §5.2 · pp. 7–8, §5 · p. 7

5.2 Results Analysis

  • Restoring parameters can improve performance, with larger gains for models fine-tuned on larger datasets or lower-mastery data.With 1,920 samples, DM train−0 gained 10.48% after restoring 40% of parameters, while DM train−4 gained a maximum 3.44% after restoring 20%.Source: §5.2 · pp. 7–8, Table 4 · p. 8
  • Restoration benefits extend to other tasks, and training-set performance can also improve, suggesting some SFT updates help neither fitting nor generalization.The magnitude of gains depends on the relevance of the task to the model’s knowledge.Source: §5.2 · p. 8, Table 5 · p. 8
  • Models trained with 1,920 samples continue improving after 40% restoration, whereas 240-sample models generally decline when restoration exceeds 20%.DM train−0 models allow more restoration and achieve greater gains than other models, regardless of dataset size.Source: §5.2 · p. 8

6 Conclusion

Across five LLMs from two model families, the paper finds that fine-tuning data category and scale shape CBQA performance. Its token- and parameter-level analyses link knowledge shifts with degraded performance and show that restoring some SFT updates can improve performance while preserving prior knowledge.

6 Conclusion

  • Across five LLMs from two families, CBQA performance varies unexpectedly with fine-tuning data category and scale.The paper evaluates models from the LLaMA-2 and LLaMA-3 families.Source: §6 · p. 9
  • Large changes in token logits correlate with degraded performance, suggesting excessive parameter updates can harm model knowledge.Source: §6 · p. 9
  • Up to 90% of SFT updates are unnecessary or detrimental for knowledge enhancement, and selectively restoring them improves performance while preserving prior knowledge.Source: §6 · p. 9

Limitations

The study does not propose an efficient fine-tuning strategy and limits its analysis to LLaMA-2 and LLaMA-3 models. Its SFT comparisons use uniform prompt templates.

Limitations

  • The study is phenomenological and does not propose an efficient fine-tuning strategy; its analysis is limited to LLaMA-2 and LLaMA-3.The authors report preliminary validation on other model families and suggest the conclusions generalize.Source: Limitations · p. 9

A Prompt for SFT

B More Details of Experiments

The experiments use five models from the LLaMA-2 and LLaMA-3 families and assess parameter restoration by ranking parameters according to their relative change after fine-tuning. The appendix also presents additional experimental results omitted from the main text due to space limitations.

B.1 Details of Models

  • Five models from two LLaMA families are analyzed: LLaMA-2-7B, LLaMA-2-13B, LLaMA-2-70B, LLaMA-3-8B, and LLaMA-3-70B.The study selects three LLaMA-2 models and two LLaMA-3 models.Source: §B.1 · p. 14

B.2 Details of Parameter Restoration

  • Parameter restoration ranks parameters by relative change between their pretrained and fine-tuned values, then measures the cumulative share of change among the highest-ranked parameters.The top 1% of parameters contribute 70.59% of total relative change.Source: §B.2 · p. 14

C More Results

  • Additional experimental results not included in the main text are presented in the appendix because of space limitations.The supplied passage introduces those results but does not specify their findings.Source: C More Results · p. 15

C.1 Test Results for the LLaMA-2 Family Models

Across the reported strategy comparisons, data division and parameter restoration enhance model performance. Additional results show performance degradation beyond a data-size point, while restoring parameters improves training-set performance.

C.1 Test Results for the LLaMA-2 Family Models

  • Performance degradation beyond a data-size point is consistent across the reported LLaMA-2 models, although peak performance occurs at different sizes.The passage states this trend is consistent despite variation by base model and hyperparameters.Source: §C.1 · p. 15

C.2 Performance on the Training Set

  • Parameter restoration improves LLaMA-3-8B performance on its training sets across the evaluated fine-tuning datasets.Table 6 reports performance after restoring different parameter proportions; improvements over the non-restored model are highlighted.Source: §C.2 · p. 16, Table 6 · p. 16

C.3 Comparison of Results Across Different Strategies

  • Data division and parameter restoration significantly enhance performance in the comparison of fine-tuning strategies.Table 7 compares LLaMA-3-8B-Instruct, mixed and divided SFT, LoRA, and parameter restoration; best results are reported for the divided and restoration strategies using 1,920 samples.Source: §C.3 · p. 16, Table 7 · p. 16

D Data Distribution of Different LLMs

The appendix reports model-specific data distributions and examines where redundant parameter updates occur and which layers are important for preserving model behavior.

D Data Distribution of Different LLMs

E.1 Distribution of Redundant Parameter Updates

  • Redundant updates are distributed across all layers, concentrated more in layers 0–2 and less in layers 30–31, with a more uniform pattern in layers 3–29.Most redundant updates occur in FFN layers; the authors suggest early layers may encode semantic information learned during pretraining, while final layers focus on output formatting.Source: §E.1 · p. 18, Table 9 · p. 18, Table 10 · p. 18

E.2 Layers for Preserving Model Behavior

  • Restoring parameters in lower layers 0–3 most affects model behavior, while middle layers 4–27 have the least impact.The experiments selectively restore parameters in different layers and evaluate resulting performance.Source: §E.2 · p. 18, Table 11 · p. 18

F Details of Data Processing

The data-processing appendix describes a multi-template procedure for measuring models’ mastery of facts and lists the topics and mappings used for in-domain and out-of-domain data.

F Details of Data Processing

  • The appendix section provides additional details on data processing.The supplied passage introduces the section without specifying further processing steps.Source: F Details of Data Processing · p. 19

F.1 Robust Multi-Template Complementation Mechanism

  • The multi-template procedure tests whether generated outputs complete a fact’s object and uses 21 mappings and 10 sampled outputs per mapping to estimate memorization.A synonym mapping table expands equivalent entity names to reduce bias from singular entity labels and is also used to judge post-SFT answers.Source: §F.1 · p. 19, Table 12 · p. 19

F.2 Topics and Mapping Templates of Data

  • The topic set covers 10 location-related in-domain topics and 14 unrelated out-of-domain topics, with 21 mapping templates designed for each topic.The appendix lists the corresponding topic data and templates in Tables 13–36.Source: §F.2 · p. 20, Table 13 · p. 20, Table 14 · p. 21
Loading 2509.16596v2…