Source-linked AI summary
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, Jonathan Herzig
TL;DR
The paper examines whether supervised fine-tuning on facts outside an LLM’s pre-existing knowledge promotes hallucinations. Using controlled closed-book QA experiments that vary Unknown-example proportions, it finds that such examples are learned slowly but, once learned, increase hallucination tendencies, while fine-tuning primarily improves use of existing knowledge.
Problem
The paper asks whether exposure to new factual information during fine-tuning affects an LLM’s ability to use pre-existing knowledge and its tendency to hallucinate.
Method
The study varies the proportion of Unknown examples in controlled closed-book QA fine-tuning datasets and evaluates how models use pre-existing knowledge.
Results
Unknown fine-tuning examples are learned slowly, and acquiring them is correlated with hallucinations relative to pre-existing knowledge.
Takeaways & Limitations
The findings highlight risks in using supervised fine-tuning to update LLM knowledge and support viewing fine-tuning as a way to improve use of pre-training knowledge.
Takeaways & Limitations
The experiments use a single LLM and closed-book QA, so variation across models and applicability to long-form generation remain unclear.
Abstract
from arXiv · showhide
When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training. It is often conjectured that this can teach the model the behavior of hallucinating factually incorrect responses, as the model is trained to generate facts that are not grounded in its pre-existing knowledge. In this work, we study the impact of such exposure to new knowledge on the capability of the fine-tuned model to utilize its pre-existing knowledge. To this end, we design a controlled setup, focused on closed-book QA, where we vary the proportion of the fine-tuning examples that introduce new knowledge. We demonstrate that large language models struggle to acquire new factual knowledge through fine-tuning, as fine-tuning examples that introduce new knowledge are learned significantly slower than those consistent with the model's knowledge. However, we also find that as the examples with new knowledge are eventually learned, they linearly increase the model's tendency to hallucinate. Taken together, our results highlight the risk in introducing new factual knowledge through fine-tuning, and support the view that large language models mostly acquire factual knowledge through pre-training, whereas fine-tuning teaches them to use it more efficiently.
1 Introduction
The paper asks whether fine-tuning on facts outside an LLM’s pre-existing knowledge increases hallucinations. In a controlled closed-book QA study, it finds that new knowledge is learned slowly but eventually increases hallucination tendencies, while fine-tuning mainly improves use of existing knowledge.
- SliCK categorizes fine-tuning examples as Known or Unknown, with Known examples further divided into HighlyKnown, MaybeKnown, and WeaklyKnown.
- In controlled closed-book QA experiments, the researchers vary the proportion of Unknown examples while controlling other factors.
- Learning from Unknown examples is linearly correlated with greater hallucination tendencies, whereas learning from Known examples is correlated with better use of pre-existing knowledge.
- Unknown examples are fitted substantially slower than Known examples, indicating that LLMs struggle to integrate new factual knowledge through fine-tuning.
- Early stopping or filtering Unknown examples can reduce overfitting risk without sacrificing performance, although filtering may remove examples useful for expressing uncertainty.
- The study isolates how fine-tuning examples containing new factual knowledge affect hallucinations relative to pre-existing knowledge.
2 Study Setup
The study constructs controlled fine-tuning datasets from factual triplets converted into closed-book question-answer pairs. It uses diverse ENTITYQUESTIONS relations, varies Unknown-example content, and evaluates exact-match performance with PaLM 2-S.
- The researchers create fine-tuned models by varying the proportion of examples unknown to the pre-trained LLM while keeping the dataset setup controlled.
- Factual knowledge is represented as subject-relation-object triplets and converted into closed-book QA pairs such as asking where Paris is located.
- Figure 2 presents formal SliCK knowledge-category definitions alongside real examples from the annotated ENTITYQUESTIONS dataset.
- ENTITYQUESTIONS supplies diverse Wikidata relations covering biographical, geographical, ownership, authorship, and historical facts.
- The experiments use PaLM 2-S and exact match as the evaluation metric.
3 Quantifying Knowledge in LLMs
The paper defines model knowledge operationally through the probability of producing a correct answer under varied prompts and decoding conditions. This continuous measure supports four SliCK categories distinguishing Unknown examples from degrees of Known knowledge.
- SliCK: SliCK uses a continuous PCorrect measure to assign each dataset pair to one of four knowledge categories.
- Defining PCorrect: PCorrect estimates how likely the model is to generate the correct answer when prompts use random few-shot exemplars and a specified decoding temperature.
- Operationalization: PCorrect is estimated with 10 random 4-shot prompts, combining greedy answers at T = 0 with sampled answers at T = 0.5.
- Known and Unknown: The study treats a question-answer pair as Known when the model sometimes predicts the correct answer and Unknown when it never does.
- Known categories: Known examples are divided into HighlyKnown, MaybeKnown, and WeaklyKnown according to degrees of certainty reflected in greedy predictions.
4 How Harmful are Unknown Examples?
Unknown examples are mostly harmless at early stopping because the model has not yet fitted them, but they become harmful with continued training. Their effect is linked to slower fitting, overfitting, and reduced accuracy, including on out-of-distribution relations.
- Overall effect: Higher %Unknown lowers performance across fine-tuning durations, while EARLY_STOP typically gives the best results.Longer training usually reduces performance, with the lowest performance observed at CONVERGENCE.
- Unknown Examples: Harmful or Neutral?: At EARLY_STOP, removing Unknown examples has almost no effect, but at CONVERGENCE their harm grows proportionally with the Unknown ratio.The comparison uses D and an ablated DKnown containing only Known examples.
- Unknown Examples: Harmful or Neutral?: The large EARLY_STOP-to-CONVERGENCE performance gap appears only when Unknown examples are present, indicating that they increase overfitting susceptibility.The gap is very small for DKnown but very large for D.
- Training dynamics: Unknown examples are fitted substantially slower than Known examples, so early stopping fits most Known examples while fitting only a small fraction of Unknown examples.This explains their neutral early-stopping effect and suggests that fine-tuning struggles to acquire new factual knowledge.
- Linear model: Fitting Unknown examples hurts test accuracy while fitting Known examples improves it, with the estimated negative and positive impacts being approximately equal.The linear model estimates coefficients from accuracy and fitted-example counts collected after each epoch across dataset variants.
- Generalization to New Relations: The same pattern transfers to out-of-distribution relations: higher Unknown ratios reduce performance, and Unknown examples are harmful mainly when fitted.The OOD linear model reports R2 = 0.95 and similar coefficient signs and magnitudes.
5 Understanding Knowledge Types: Their Value and Impact
Fine-grained knowledge categories reveal that dataset composition matters for test performance. MaybeKnown examples provide the best overall results, while WeaklyKnown and Unknown examples become more prone to overfitting with continued training.
- Experimental setup: Single-category dataset variants isolate how each knowledge category affects overall and per-category test performance.The variants keep |D| fixed, and DNatural preserves the natural ENTITYQUESTIONS category distribution.
- Limited Knowledge Enhances Overfitting: WeaklyKnown and Unknown datasets suffer substantial performance drops from EARLY_STOP to CONVERGENCE: 39.2 →35.4 and 37.5 →25.8, respectively.Longer training modestly improves performance on their own categories but degrades performance on HighlyKnown and MaybeKnown.
- MaybeKnown Examples are Essential: DNatural matches DMaybeKnown at EARLY_STOP but degrades significantly by CONVERGENCE, likely because its WeaklyKnown and Unknown examples promote overfitting.The presence of MaybeKnown examples appears sufficient for high MaybeKnown performance at early stopping.
6 SliCK Knowledge Categories Analysis
This section evaluates SliCK’s four-category taxonomy and compares its Unknown classification with a P(True)-threshold approach. The analyses indicate that SliCK’s fine-grained Known categories distinguish knowledge strength, while its Unknown examples are generally not known to the model.
- Taxonomy significance: SliCK is proposed as a useful taxonomy for guiding future research on assessing what LLMs know.The authors note that evaluating knowledge-assessment methods remains challenging because there is no ground truth for what the model truly knows.
- Fine-grained Known Categories: HighlyKnown examples consistently exceed 95% post-fine-tuning accuracy, while WeaklyKnown accuracy falls below MaybeKnown accuracy but remains above Unknown accuracy.The category distinctions also revealed insights about the importance of MaybeKnown fine-tuning examples.
- Comparison with P(True): Figure 5 compares SliCK’s Unknown category with classifying examples as Unknown when P(True) falls below a threshold.The figure plots the percentage classified as Unknown against post-fine-tuning accuracy on those examples.
- Benchmarking Unknown Test Examples: Unknown test examples achieve at most 3.2% accuracy post-fine-tuning, indicating that most are genuinely unknown to the model.Because the closed-book train and test sets are disjoint, answering test questions requires reliance on pre-existing knowledge.
- Limitation: A comprehensive comparison of SliCK with alternative approaches remains future work because the presented analysis is preliminary.The limitation is explicitly framed as a preliminary analysis rather than a completed comparison.
7 Fine-tuning to Abstain on Unknown Examples
This section tests whether relabeling Unknown fine-tuning examples as “I don’t know” can reduce overfitting. The experiment suggests that uncertainty relabeling lowers the risk while preserving higher accuracy among questions the model answers.
- Practical mitigation: Fitting Unknown examples negatively affects test performance, but early stopping or filtering Unknown examples can mitigate this overfitting.The negative effect is described as emerging through later-stage overfitting.
- Approach: Relabeling Unknown fine-tuning examples as “I don’t know” is tested as a mitigation for the observed overfitting.The model is fine-tuned after replacing the labels of Unknown examples with the uncertainty expression.
- Results: Uncertainty relabeling appears to reduce the risk of overfitting, while accuracy improves on willingly answered test examples.Table 3 measures accuracy only on questions that received an answer rather than “I don’t know”.
8 Discussion
The discussion frames Unknown-example fine-tuning as a practical hallucination risk and as evidence that fine-tuning mainly affects use of pre-existing knowledge. It presents early stopping, filtering, and uncertainty relabeling as mitigation strategies, while noting trade-offs and task-dependent stopping points.
- Practical Implications: Fine-tuning on new factual knowledge is correlated with hallucinations relative to pre-existing knowledge, highlighting a risk in supervised fine-tuning for knowledge updates.The discussion connects this risk to the slower fitting of Unknown examples and their overfitting-like negative effect.
- Mitigation trade-offs: Early stopping is emphasized because Unknown examples fit more slowly, although it may be less effective across numerous tasks with distinct optimal stopping points.The authors contrast early stopping with aligning fine-tuning data to the model’s existing knowledge by filtering Unknown examples.
- Filtering and uncertainty: Filtering Unknown examples can reduce overfitting without compromising performance, but may remove examples useful for teaching uncertainty expression.This trade-off motivates testing uncertainty relabeling as an alternative.
- Relabeling: Relabeling Unknown examples with uncertainty expressions appears to reduce their negative effect and may be the most promising approach among those discussed.The conclusion is based on the experiment described in Section 7.
- Superficial Alignment Hypothesis: The findings support pre-training as the main source of factual knowledge while showing that fine-tuning example selection affects utilization of that knowledge.Fine-tuning is therefore described as teaching more than interaction style or format.
9 Related Work
Related work covers the conjecture that fine-tuning on new knowledge can cause hallucinations, evidence from factuality studies, behavior on unknown information, and methods for quantifying model knowledge. SliCK is positioned among confidence-elicitation approaches that assess knowledge of ground-truth answers.
- New knowledge and hallucinations: Prior work conjectures that fine-tuning on new factual knowledge may encourage hallucinations, a scenario also described as capability misalignment.The related work notes that limited research addresses this scenario because defining an LLM’s knowledge boundary is difficult.
- Factuality under fine-tuning: Models fine-tuned on well-known facts show enhanced factuality compared with models fine-tuned on unpopular facts, while retrieval-augmented training data has been associated with reduced factuality.The discussion attributes these patterns to differences in model familiarity with the facts.
- New knowledge at test time: At test time, models may mimic responses associated with unknown fine-tuning examples, while performance remains unsatisfactory when new knowledge appears in input contexts.Related work also analyzes unknown examples in in-context learning.
- Quantifying knowledge: SliCK treats knowledge assessment as confidence elicitation for ground-truth labels, complementing methods based on sampling agreement, internal representations, verbalized probability, and prompting.The method considers the model to know a question-answer pair when it is confident that the answer is correct.
10 Conclusion
The paper studies how introducing new factual knowledge through fine-tuning affects hallucinations and finds that models struggle to integrate it while mostly learning to use pre-existing knowledge.
- The study isolates how new factual knowledge introduced through fine-tuning affects the model’s tendency to hallucinate.
- Acquiring new knowledge through supervised fine-tuning is correlated with hallucinations relative to the model’s pre-existing knowledge.
- LLMs struggle to integrate new knowledge through fine-tuning and mostly learn to use their pre-existing knowledge.
11 Limitations
The study’s conclusions are limited by its single-model, closed-book QA setting, approximated knowledge categorization, full-fine-tuning regime, and carefully controlled dataset design.
- The experiments use a single LLM, so it remains unclear whether the results vary across different models.
- The practical implications of filtering Unknown examples require validation beyond closed-book QA, especially in long-form text generation.
- Adding diverse tasks to the fine-tuning mixture was not tested because it could introduce new factual knowledge uncontrollably.
- The study focuses on full fine-tuning, leaving parameter-efficient regimes such as LoRA for future research.
- SliCK only approximates the LLM’s knowledge, although the authors report that misclassification occurs in at most 3% of cases.
I Out-of-distribution (OOD) Evaluation
The OOD evaluation uses seven relations unseen during fine-tuning and finds similar trends to in-distribution evaluation, but with smaller performance drops.
- The OOD test set contains seven relations unseen during fine-tuning.
- The statistical analysis uses paired-sample t-tests over 100 approximately equally sized shuffled subsets, with thresholds p < 0.05 and p < 0.01.
- For nearly all Table 2 comparisons, differences are statistically significant at p < 0.01, with two exceptions significant at p < 0.05.
- The EARLY_STOP versus CONVERGENCE difference is significant for all comparisons except DMaybeKnown.