PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Fine-Tuning Encourages Hallucinations and How to Fix It

Guy Kaplan, Zorik Gekhman, Zhen Zhu, Lotem Rozner, Yuval Reif, Swabha Swayamdipta, Derek Hoiem, Roy Schwartz

arXiv:2604.15574v1cs.CLcs.AIcs.LGcs.NE

TL;DR

Supervised fine-tuning can cause models to hallucinate about facts they previously knew, raising the challenge of preserving factual stability while learning new information. This paper applies continual-learning strategies and finds that self-distillation reduces forgetting from approximately 15% to 3% while retaining factual plasticity, with interference among overlapping representations as a primary mechanism.

  • Problem

    Supervised fine-tuning can impair previously acquired factual knowledge, creating a need to learn new facts without increasing hallucinations.

  • Method

    The paper evaluates selective parameter freezing when new fact acquisition is unnecessary and self-distillation to constrain output-distribution drift when it is required.

  • Results

    Self-distillation reduces factual forgetting from ∼15% to ∼3% while preserving factual plasticity, and forgetting is linked to interference among overlapping semantic representations.

  • Takeaways & Limitations

    SFT-induced hallucinations are not an inevitable cost of factual learning and can be mitigated by targeting representational interference.

Abstract

from arXiv · show

Large language models are prone to hallucinating factually incorrect statements. A key source of these errors is exposure to new factual information through supervised fine-tuning (SFT), which can increase hallucinations w.r.t. knowledge acquired during pre-training. In this work, we explore whether SFT-induced hallucinations can be mitigated using established tools from the continual learning literature, since they arise as a by-product of knowledge degradation during training. We propose a self-distillation-based SFT method that facilitates effective factual learning while minimizing hallucinations w.r.t. pre-existing knowledge by regularizing output-distribution drift. We also show that, in settings where new knowledge acquisition is unnecessary, suppressing factual plasticity by freezing parameter groups, can preserve task performance while reducing hallucinations. Lastly, we investigate the mechanism behind SFT-induced hallucinations through three hypotheses: capacity limitations, behavior cloning, and localized interference. Our experiments show that a main driver is interference among overlapping semantic representations, and that self-distillation succeeds by mitigating this interference.

1 Introduction

The paper frames SFT-induced hallucinations as factual forgetting caused by continual-learning dynamics, and proposes selective freezing or self-distillation to reduce them. Its experiments attribute the errors primarily to interference among overlapping semantic representations.

  • SFT can make models answer previously correct factual questions incorrectly after learning new factual knowledge, worsening a reliability challenge.The paper connects this phenomenon to factual forgetting during fine-tuning and distinguishes it from pre-training knowledge gaps or inference-time hallucinations.
  • Freezing selected parameter groups reduces factual plasticity, enabling downstream-task learning while limiting new fact acquisition and hallucinations.This mitigation is intended for settings where acquiring new facts is undesirable, such as private-domain SFT or alignment fine-tuning.
  • Self-distillation regularizes the fine-tuned model toward its earlier output distribution, reducing SFT-induced hallucinations while supporting acquisition of newly introduced facts.Unlike freezing, this approach preserves effective factual learning rather than deliberately suppressing factual plasticity.
  • The proposed mechanism is selective factual forgetting caused by interference among overlapping semantic representations, rather than only capacity limits or behavior cloning.The study tests these hypotheses using synthetic facts while varying entity-name scale and surface form.

2 Fine-Tuning with Unknown Facts Leads to Factual Forgetting

This section reframes SFT-induced factual hallucinations as factual forgetting and separates task learning from factual learning. It establishes a controlled evaluation using SLiCK, ENTITYQUESTIONS, and mixtures of Known and Unknown facts.

  • Motivation: SFT can increase factual hallucinations, which the paper interprets as factual forgetting through a continual-learning lens.The experimental setting separates learning how to perform a task from learning facts.
  • Knowledge Classification: SLiCK classifies each question as HighlyKnown, MaybeKnown, WeaklyKnown, or Unknown using predictions from randomized few-shot prompting configurations.HighlyKnown requires consistently correct answers, whereas Unknown requires no correct answers across configurations.
  • Data: ENTITYQUESTIONS provides Wikipedia-derived question–answer pairs, with 20 SLiCK evaluation runs per question and retention of relations having at least 30% HighlyKnown examples.The benchmark covers diverse entity–relation types, including questions such as France’s capital.
  • Models: Experiments use QWEN 2.5 models with 1.5B and 8B parameters and LLAMA 3.1 with 8B parameters, primarily reporting QWEN 2.5-1.5B results.The paper states that qualitative trends are consistent across models.
  • Training Procedure: Models are fine-tuned at learning rate 5 × 10−5 on mixtures of Known and Unknown facts using either a 50/50 ratio or Only Known with DUnk = ∅.Performance is evaluated separately on DKnown, DUnk, and DHeld for task learning, factual plasticity, and factual stability.

3 Factual Forgetting and the Factual Plasticity–Stability Tradeoff in SFT

Standard SFT first learns the QA task, then acquires new facts while degrading previously held knowledge. Selectively freezing modules—especially updating attention layers only—can retain task learning, suppress factual plasticity, and reduce hallucinations when new facts are unnecessary.

  • Parameter freezing: Updating only attention layers achieves high DHeld and DKnown while suppressing new fact acquisition, with DUnk ≈0.This parameter-freezing strategy reduces factual forgetting while retaining task learning.
  • Standard SFT dynamics: Standard SFT rapidly learns the QA format, then acquires new facts as performance on previously held knowledge declines.The first one to two epochs yield near-perfect DKnown accuracy and 93% DHeld accuracy; later DUnk learning coincides with DHeld decline.
  • Standard SFT dynamics: Once the model begins integrating new facts, standard fine-tuning induces factual forgetting.The Only Known condition provides a baseline without forgetting caused by factual plasticity.
  • Parameter freezing: Training only FFN layers preserves the coupling between factual plasticity and forgetting, closely tracking full-model SFT.The model continues learning the task and new facts while forgetting on DHeld follows the full-model pattern.
  • Key insight: Reducing factual plasticity suppresses hallucinations while retaining task learning.The strategy is intended for settings where incorporating new facts is unnecessary or undesirable.

4 Enabling Factual Learning without Forgetting Facts

Self-distillation enables models to learn new factual knowledge while preserving previously stored facts. By constraining output-distribution drift, it reduces SFT-induced forgetting and hallucinations without suppressing factual plasticity.

  • Motivation: Self-distillation addresses the stability–plasticity dilemma by enabling new factual learning while preserving previously stored knowledge.The method targets settings where acquiring new facts is necessary and cannot simply be avoided.
  • Method: Self-distillation constrains shifts in the model’s output distribution to limit training-induced drift and protect previously learned factual knowledge.A frozen teacher snapshot regularizes the student while still allowing factual learning.
  • Results: Self-distillation enables factual learning at a pace comparable to standard fine-tuning while largely preserving held-out factual performance.Performance on DHeld closely matches the Only Known condition, while accuracy on DUnk reaches a level comparable to regular SFT.
  • Results: Approximately 3% decline on DHeld under self-distillation compares with a 15% decline under regular SFT, indicating substantially reduced forgetting.Performance on DUnk follows a similar learning trajectory to standard fine-tuning, indicating high factual plasticity.
  • Conclusion: Self-distillation reduces SFT-induced hallucinations.This is identified as the section’s central result for fine-tuning that requires both factual learning and knowledge preservation.

5 Semantic Overlap Drives Interference in Factual Updates

Controlled synthetic updates show that SFT-induced factual forgetting is driven primarily by interference among semantically overlapping entity representations, not global capacity limits. Self-distillation suppresses this interference and preserves representations of previously known entities while enabling factual acquisition.

  • Controlled synthetic updates: Across scales, models acquired all new facts (DUnk = 100%) and retained high DKnown performance (≥90%), isolating induced forgetting as the key difference.The analysis therefore focuses on ∆DHeld, the percentage-point drop from peak held-out accuracy during training.
  • Semantic overlap drives forgetting: For semantically similar name-like keys, forgetting grew sharply with added facts, whereas unrelated UUID keys showed negligible forgetting (0–4%) across scales.This asymmetric scaling pattern contradicts a simple global capacity constraint and supports selective interference among nearby representations.
  • Why self-distillation works: Generic ℓ2 regularization left forgetting near 10 percentage points, while stronger regularization impaired acquisition, so weight movement alone does not explain self-distillation’s benefit.The comparison matched ℓ2 regularization to self-distillation in gradient magnitude.
  • Representation-level mechanism: Hidden-state drift rose to ≈5% during initial task-format learning in all conditions, then reached ≈11% for semantic-key SFT but stabilized near ≈5% for UUID SFT and self-distillation.Drift was measured using cosine distance at layer 14, the middle layer of the 28-layer QWEN2.5 model.
  • Mechanistic conclusion: Together, the results identify localized interference among semantically overlapping entity representations—not generic weight drift or behavioral effects—as the origin of SFT-induced hallucinations.Self-distillation reduces forgetting by constraining output distributions over overlapping entity tokens, preventing excess representational drift.

6 Related Work

Related work frames hallucinations as unfaithful outputs arising across data, training, and inference, while showing that factual knowledge is distributed across model components and may remain hidden during recall failures. Continual-learning methods address the stability–plasticity tension through regularization, replay, architectural isolation, and distillation.

  • Hallucinations and scope: Hallucinations are commonly defined as outputs unfaithful to real-world facts, provided context, or user instructions, arising from mechanisms spanning data, training, and inference.
  • Factual knowledge in LLMs and module roles: Prior studies find that both FFN layers and attention projections contribute to storing and expressing factual knowledge, suggesting representations are distributed across architectural components.
  • Hidden knowledge and recall failures: LLMs can hallucinate while internally retaining correct answers, indicating that some failures reflect degraded recall rather than complete knowledge loss.
  • Continual learning in LLMs: Continual-learning research studies the stability–plasticity tension in sequential training and proposes regularization, replay, architectural isolation, and distillation as mitigations.

7 Conclusion

The conclusion reframes SFT-induced hallucinations as factual forgetting driven by continual-learning dynamics and representational interference, rather than an inevitable cost of factual learning. It presents selective FFN freezing and self-distillation as effective mitigation strategies while motivating factual stability as a first-class fine-tuning objective.

  • Mitigation strategies: SFT-induced hallucinations are framed as factual forgetting arising from continual-learning dynamics, with established mitigation strategies transferring effectively.The conclusion distinguishes settings where new fact acquisition is undesirable from those where it is required.
  • Mitigation strategies: Selective FFN freezing suppresses factual plasticity when new fact acquisition is undesirable while preserving task learning.This intervention targets factual plasticity without eliminating task learning.
  • Mitigation strategies: Self-distillation constrains output-distribution drift when new fact acquisition is required, reducing forgetting from ∼15% to ∼3% without sacrificing factual plasticity.The method supports factual learning while limiting degradation of pre-existing knowledge.
  • Mechanism: Forgetting scales with representational overlap for semantically similar facts, but drops to near-zero for random UUID facts even at 106 new facts.This selective asymmetry is difficult to reconcile with capacity or behavioral accounts.
  • Mechanism: The findings identify localized interference among overlapping representations as a primary driver of SFT-induced hallucinations.The unified view treats representational interference as a target for direct mitigation rather than an unavoidable consequence of factual learning.
  • Implications: The conclusion encourages treating factual stability as a first-class objective in fine-tuning and leveraging continual learning to achieve it.This framing motivates future work on stability-aware fine-tuning.

A Results Over Other SLiCK Classification Groups

Across SLiCK knowledge groups, fine-tuning effects differ sharply: Regular SFT most strongly forgets pre-existing knowledge in HighlyKnown and MaybeKnown facts, while WeaklyKnown facts show an inverted ordering. Self-distillation stabilizes HighlyKnown accuracy, whereas Only Known preserves it but performs poorly on WeaklyKnown facts.

  • Knowledge-group composition: 46.6% of the 3,530 development facts are HighlyKnown, while MaybeKnown facts comprise 32.4%, with the four selected relations each exceeding 30% HighlyKnown.WeaklyKnown and Unknown facts account for 5.3% and 15.8%, respectively.
  • HighlyKnown: On held-out HighlyKnown facts, Only Known maintains ≈95% accuracy, self-distillation stabilizes at ≈92%, and Regular SFT degrades to ≈80%.Regular SFT is approximately 13 percentage points below the pre-fine-tuning baseline.
  • MaybeKnown: Regular SFT causes a larger MaybeKnown collapse, reducing accuracy from approximately 0.64 to 0.43, a drop of roughly 21 percentage points.MaybeKnown facts are less robustly encoded because correctness occurs under only some SLiCK prompt configurations, increasing susceptibility to representational interference.
  • WeaklyKnown: WeaklyKnown facts invert the ordering: baseline accuracy is ≈0.15–0.17, while Only Known drops to approximately 0.04–0.05 and Regular SFT remains around ≈0.14.These facts contain highly latent knowledge and are never correctly predicted as top-1 under any SLiCK configuration.

B Results Across Different Models

Across Llama-3.1-8B and Qwen2.5-7B, the paper’s core training dynamics replicate those observed in the primary experiments. Self-distillation mitigates forgetting across architectures, while fine-grained parameter updates and partially known facts reveal additional structure.

  • Cross-model replication: The core training dynamics replicate across Llama-3.1-8B and Qwen2.5-7B, consistent in magnitude with the primary 1.5B-parameter experiments.Regular SFT rapidly reaches 97–98% accuracy on DKnown before DHeld progressively falls to approximately 85–86%.
  • Cross-model replication: Self-distillation stabilizes DHeld accuracy at approximately 95%, versus 95–97% under Only Known and an 11–12 percentage-point drop under regular SFT.The residual self-distillation degradation is approximately 2–3 percentage points, corresponding to approximately 80% mitigation of forgetting.
  • Fine-grained freezing: Forgetting mitigation requires restricting updates to a single attention projection, such as k, v, or o, rather than training the full attention block.This shows that update granularity within attention matters beyond the coarse attention-versus-FFN distinction.
  • MaybeKnown facts: Regular SFT reduces MaybeKnown accuracy by approximately 18–19 percentage points for Llama-3.1-8B, from ∼0.63 to ∼0.45.The effect is consistent across both model families and sizes and is substantially more pronounced than for DHeld.

C Results Over Different Module Freezing Experiments

Fine-grained freezing experiments show that forgetting is governed by factual plasticity, measured by DUnk, rather than by which parameter group is updated. Attention projections suppress plasticity most strongly, while gate+up induces more plasticity than down within the FFN.

  • Stability–plasticity tradeoff: Freezing is one way to control factual integration, while self-distillation provides another stability mechanism without sacrificing plasticity.The results attribute reduced forgetting to decreased DUnk rather than to freezing itself.
  • Attention projections: Individual attention projections suppress plasticity most aggressively, with DUnk ≈0.005–0.006, and yield the highest DHeld values.The projections evaluated are q, k, v, and o.
  • Mechanism of forgetting: DHeld decreases monotonically with DUnk across parameter-freezing configurations, showing that factual plasticity mediates forgetting.The relationship holds irrespective of which parameter group is updated.
  • FFN sub-components: Within the FFN, gate+up drives substantially more plasticity than down.The comparison comes from fine-grained freezing ablations across FFN sub-components.

D Results Over Different Parameters for Self-distillation · E Synthetic Entities Creation Procedure

Self-distillation performs best with a balanced regularization weight and low temperature, while extreme snapshot and hyperparameter choices trade factual acquisition speed against stability. Synthetic-entity experiments isolate key semantics as the variable driving forgetting by holding the relation fixed and varying key and value types.

  • D Results Over Different Parameters for Self-distillation: Self-distillation varies snapshot epoch i, distillation weight λ, and temperature τ, with ablations evaluating DUnk factual plasticity and DHeld factual stability.Figure 9 fixes the other two parameters at λ = 1 and τ = 0.5 for each independent ablation.
  • D Results Over Different Parameters for Self-distillation: Freezing the teacher at i = 0 substantially slows DUnk acquisition and produces the lowest DHeld across snapshot choices.The distillation loss constrains the student toward a pre-adaptation parameter space, while lower DHeld may also reflect difficulty learning the task format.
  • E Synthetic Entities Creation Procedure: Semantic keys recombine position-aware tokens from real place names into novel two- or three-token names, discarding candidates that match existing places.For example, Bergamo and Pasadena yield Bergadena, which shares sub-word structure with genuine locations.
  • E Synthetic Entities Creation Procedure: UUID keys use random 8-character hex identifiers without lexical overlap with real entities, providing a contrast to semantically structured keys.Both key conditions are paired with real country names under an identical question setup.
  • D Results Over Different Parameters for Self-distillation: Weaker regularization λ = 0.1 or λ = 0.5 accelerates DUnk acquisition but increases DHeld degradation, whereas λ = 1.5 reduces forgetting while delaying factual learning.The selected λ = 1 balances efficient factual acquisition with meaningful forgetting mitigation.
  • D Results Over Different Parameters for Self-distillation: Higher temperatures τ = 2 and τ = 4 smooth DUnk learning trajectories but lower DHeld, while τ = 0.5 and τ = 1 preserve better stability with comparable final DUnk accuracy.The default is τ = 0.5; higher temperatures may suit users prioritizing stable learning trajectories despite additional forgetting.
  • E Synthetic Entities Creation Procedure: Controlled experiments fix the P17 location→country relation, question template, answer domain, and relational structure, making entity names the sole experimental variable.This design is intended to make comparisons between key types causally clean.
  • E Synthetic Entities Creation Procedure: Across real, semantic, and UUID country values, semantic keys consistently induce 38–41 percentage-point ΔDHeld after training on 106 synthetic facts.All combinations achieve DUnk = 100% and DKnown ≥90%, indicating that forgetting differences are not explained by unequal learning success.

F Further Analyses on Self-Distillation

Further analyses show that constraining the teacher’s high-probability output region, rather than an arbitrary number of logits, preserves held-out knowledge during factual learning. Six drift metrics consistently indicate that self-distillation reduces the representational and output-space interference induced by SFT while maintaining comparable acquisition of new facts.

  • Output-distribution ablations: Top-k self-distillation matches full self-distillation, reducing DHeld forgetting to ≈3 percentage points while DUnk acquisition proceeds at the same pace.The 76 selected tokens capture 91%–92% of the teacher’s probability mass per position.
  • Output-distribution ablations: Random-k distillation leaves forgetting at ≈10 percentage points, showing that constraining the teacher’s high-probability region matters more than the number of constrained logits.Random-k uses the same 76-token count but selects tokens uniformly, removing alignment with the teacher’s high-probability region.
  • Drift metrics: Under SFT, hidden-state drift reaches ≈11% and neighborhood restructuring reaches ≈0.07, versus ≈5% and ≈0.02 under self-distillation.Elevated RD and NS under SFT indicate that held-out representations genuinely reorganize rather than merely translating together.
  • Drift metrics: SFT raises Rank-ρ from ≈0.32 to ≈0.40 and lowers NTR from ≈45% to ≈17%, whereas self-distillation keeps Rank-ρ near ≈0.32 and NTR near 36%.These output-level shifts indicate that SFT separates new-entity outputs while increasing convergence in candidate rankings relative to held-out outputs.
  • Drift metrics: JSD rises to ≈0.68 under SFT versus ≈0.63 under self-distillation, while both conditions acquire DUnk facts comparably and self-distillation better preserves DHeld accuracy.Across all six metrics, the results support representational and output-space interference as the consistent pattern associated with SFT.
Loading 2604.15574v1…