PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Fine-Tuning Encourages Hallucinations and How to Fix It

Guy Kaplan, Zorik Gekhman, Zhen Zhu, Lotem Rozner, Yuval Reif, Swabha Swayamdipta, Derek Hoiem, Roy Schwartz

arXiv:2604.15574v2cs.CLcs.AIcs.LGcs.NE

TL;DR

Supervised fine-tuning can induce hallucinations by disrupting previously learned facts, raising questions about how to mitigate and explain this forgetting. The paper uses selective freezing and self-distillation, finding that self-distillation reduces hallucinations during new-fact learning and that forgetting tracks representational overlap.

  • Problem

    The work examines how to mitigate and explain hallucinations caused by factual forgetting during supervised fine-tuning.

    Source: §1 · p. 2

  • Method

    It applies selective parameter freezing when new facts are unnecessary and self-distillation to constrain output-distribution drift when new facts are desired.

    Source: §1 · p. 2

  • Results

    Both mitigations reduce factual forgetting from ∼15 to 2 pp, while forgetting scales with representational overlap and is near-zero for random UUID facts even at 10^6 new facts.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    SFT-induced hallucinations can be targeted as representational interference rather than treated as an inevitable cost of factual learning.

    Source: §7 · p. 10

    The associative-memory account is qualitative and relies on a first-order update, fixed key map, idealized answer geometry, and Gaussian fluctuation approximation.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Large language models are prone to hallucinating factually incorrect statements. A key source of these errors is exposure to new factual information through supervised fine-tuning (SFT), which can increase hallucinations w.r.t.~knowledge acquired during pre-training. Since these errors arise as a by-product of knowledge degradation, we explore whether established continual learning tools can mitigate them. We propose a self-distillation-based SFT method that facilitates effective factual learning while minimizing hallucinations w.r.t.~pre-existing knowledge by regularizing output-distribution drift. We also show that when new knowledge acquisition is unnecessary, suppressing factual plasticity by freezing parameter groups preserves task performance while reducing hallucinations. Lastly, we investigate the mechanism, contrasting capacity limitations, behavior cloning, and localized interference. Our experiments show that a main driver is interference among overlapping semantic representations, which self-distillation mitigates and an associative-memory model explains: forgetting grows with the overlap between new and stored facts.

1 Introduction

The paper frames SFT-induced hallucinations as factual forgetting caused by interference with pre-trained knowledge, and evaluates continual-learning mitigations that preserve existing facts while enabling task learning and new knowledge acquisition.

Motivation

  • SFT-induced hallucinations occur when models learn new factual knowledge and then answer incorrectly questions they previously answered correctly (Gekhman et al., 2024; Kalai et al., 2025).The paper investigates whether continual-learning tools can mitigate this stability–plasticity tradeoff.Source: §1 · p. 2

Mechanism

  • The paper attributes hallucinations to interference among overlapping semantic representations, rather than global capacity limits or behavior cloning.This reframes SFT-induced errors as factual forgetting distinct from pre-training knowledge gaps or inference-time hallucinations.Source: §1 · p. 2
  • Name-like entities cause forgetting that grows sharply with scale, whereas random UUID-style identifiers produce near-zero forgetting even at 10^6 new facts.Self-distillation prevents representational drift in held-out entities, and an associative-memory account explains why representational overlap governs forgetting.Source: §1 · p. 2

Mitigations

  • Selective parameter freezing and self-distillation reduce factual forgetting from ∼15 to 2 pp, with self-distillation preserving new-fact acquisition and transferring to Alpaca instruction tuning.Freezing is suited to settings where new factual acquisition is unnecessary, whereas self-distillation regularizes output-distribution drift when acquisition is desired.Source: §1 · p. 2

Factual forgetting

  • Forgetting arises from acquiring new facts rather than fine-tuning itself: held-out accuracy declines after unknown-fact acquisition but remains stable when only known facts are used.The model first learns the question-answering format before unknown facts are introduced.Source: §1 · p. 2, Figure 2 · p. 3

2 Fine-Tuning with Unknown Facts Leads to Factual Forgetting

The section reframes fine-tuning-induced factual hallucinations as factual forgetting and establishes a controlled setting that separates task learning, new-fact acquisition, and stability of pre-existing knowledge.

2 Fine-Tuning with Unknown Facts Leads to Factual Forgetting

  • SFT can increase factual hallucinations, which the study reinterprets as factual forgetting and investigates by disentangling task learning from factual learning.The controlled reconstruction follows Gekhman et al. (2024).Source: §2 · p. 3

2.1 Preliminary: SLiCK Method and Factual Learning Setting

  • SLiCK filters questions into HighlyKnown and Unknown facts, using DKnown for task learning, DUnk for factual plasticity, and DHeld for factual stability.Accuracy on these sparse, semantically isolated splits reflects distinct behavioral dimensions: adaptation to known facts, acquisition of unknown facts, and retention of pre-existing knowledge.Source: §2.1 · p. 3

2.2 Methodology and Experimental Setup

  • Experiments use ENTITYQUESTIONS with 8,000 DKnown and 8,000 DUnk training examples, a HighlyKnown DHeld development split, and QWEN 2.5 and LLAMA 3.1 models.Training uses learning rate 5 × 10−5, with either a 50/50 Known/Unknown mix or Only Known training, evaluating DKnown, DUnk, and DHeld throughout.Source: §2.2 · pp. 3–4

3 Factual Forgetting and the Factual Stability-Plasticity Tradeoff in SFT

Standard SFT causes factual forgetting when it acquires new facts, whereas restricting parameter updates can preserve pretrained factual knowledge while retaining task learning. Updating only attention layers suppresses new fact acquisition and reduces hallucinations, while FFN-only training behaves like standard SFT.

3 Factual Forgetting and the Factual Stability-Plasticity Tradeoff in SFT

  • Selective freezing separates task and factual plasticity, supporting QA-format adaptation while limiting factual updates when acquiring new knowledge is unnecessary or undesirable.The regime is motivated for settings such as alignment or privacy, where preserving pretrained knowledge may be preferable to learning new facts.Source: §3 · p. 4, §3.2 · p. 4

3.2 Controlling Factual Plasticity via Parameter Constraints

  • Updating only attention layers preserves task learning and pretrained factual performance by keeping DUnk ≈0, whereas FFN-only training closely tracks standard SFT.Table 1 reports high DHeld and DKnown for attention-only training; DKnown reaches 0.95 versus 0.99 for the Only Known upper bound.Source: Table 1 · p. 4, §3.2 · p. 4

3.1 Learning New Facts Induces Factual Forgetting and Hallucinations

  • Standard SFT’s hallucinations emerge after new-fact acquisition: DHeld peaks at 93% before declining by approximately 15 pp as DUnk rises, while Only Known remains stable.The stable Only Known condition shows that factual learning, rather than fine-tuning alone, causally drives forgetting; the decline saturates when DUnk learning converges.Source: §3.1 · p. 4

4 Enabling Factual Learning without Forgetting Facts

The section presents self-distillation as a way to acquire new factual knowledge while limiting forgetting by constraining output-distribution drift. It also shows that these mitigations transfer to realistic instruction tuning, preserving alignment quality while reducing factual degradation.

4.1 Self-Distillation Enables Fine-Tuning with Minimal Induced Hallucinations

  • Self-distillation constrains output-distribution drift toward a lightly adapted teacher while continuing SFT on known and unknown facts.The method trains on DKnown for one epoch, freezes that model as teacher, then fine-tunes on DKnown ∪ DUnk with λ = 1 and τ = 0.5.Source: §4.1 · p. 5
  • Self-distillation reduces held-out factual degradation to approximately 2 pp versus 15 pp for regular SFT while keeping the final unknown-fact acquisition gap below 1.5 pp.It learns new facts at a pace comparable to standard fine-tuning while substantially reducing forgetting.Source: §4.1 · p. 5
  • No parameter-space alternative approaches self-distillation: alternatives leave final DHeld at 0.78–0.84, whereas self-distillation preserves 0.91 at comparable factual acquisition.The comparison includes snapshot ℓ2 regularization, EWC, LoRA, and AdamW weight decay.Source: §4.1 · p. 6, Table 2 · p. 6

4.2 Mitigations Transfer to Real Instruction-Tuning Data

  • On implicit factual content in Alpaca instruction tuning, standard SFT loses 7.4 pp from peak factual accuracy, versus 1.7 pp for self-distillation and 0.9 pp for attention-only training, while alignment quality is preserved.Self-distillation slightly exceeds standard SFT on AlpacaEval; freezing provides the strongest factual stability at a modest alignment cost while still improving over the base model.Source: §4.2 · p. 6, Table 2 · p. 6

5 Semantic Overlap Drives Interference in Factual Updates

Experiments show that factual forgetting is driven primarily by interference between semantically overlapping representations, not scale or answerability alone. Self-distillation suppresses this overlap-induced drift, while an associative-memory account predicts forgetting from the interaction of representational overlap and the number of updates.

5.1 Semantic Overlap and Scale as Sources of Interference

  • At 10^6 facts, name-like keys reach 39 pp forgetting, whereas UUID keys remain at 0–4 pp despite identical supervision and complete acquisition.This asymmetric scaling is inconsistent with purely behavioral or global-capacity explanations and supports selective structural interference.Source: §5.1 · p. 7, Figure 4 · p. 7
  • At 10^4 facts, forgetting rises with pretrained key overlap: 2.1 pp for UUID, 3.8 for 2-noise, 7.2 for 1-noise, and 10.3 for token-mix.The overlap ordering already exists before fine-tuning, showing that forgetting follows a continuous dose-response rather than a binary name-like effect.Source: §5.1 · pp. 7–8

5.2 What Self-Distillation Reveals About the Origins of Factual Forgetting

  • Self-distillation suppresses overlap-induced forgetting, whereas matched ℓ2 weight regularization leaves forgetting near 10 pp and impairs acquisition when strengthened.Hidden-state drift stabilizes near ≈5% under self-distillation or UUID keys but reaches ≈11% under SFT on semantic keys, isolating the interference component.Source: §5.2 · p. 8

5.3 Why Overlap Predicts Forgetting: An Associative-Memory Account

  • The associative-memory account predicts forgetting from the product of update scale and mean squared key overlap, explaining scale, overlap, value-independence, and acquisition-related saturation.It explains why output-space regularization succeeds where weight-space regularization fails, but remains qualitative under first-order, fixed-key, idealized-geometry, and Gaussian assumptions.Source: §5.3 · pp. 9–10

6 Related Work

The paper frames SFT-induced hallucinations as factual forgetting caused by representational interference, connecting this account to distributed factual storage, hidden knowledge, and continual-learning methods. Its self-distillation approach differs from prior distillation by preserving pretrained facts rather than teaching new behavior.

Hallucinations and scope

  • SFT-induced hallucinations are treated as failures to produce previously known facts, arising when fine-tuning disturbs prior knowledge through representational interference.This framing applies to closed-book question answering, where an unrecalled previously known fact counts as a hallucination.Source: §6 · p. 10

Factual knowledge in LLMs and module roles

  • Factual associations are distributed across FFN layers and attention projections, so updating only parameter subsets can preserve skill performance while changing outcomes.The paper notes that parameter-selection effects are examined in §3.Source: §6 · p. 10

Hidden knowledge and recall failures

  • Some hallucinations reflect degraded recall rather than erased knowledge, because models may internally retain answers they fail to express; self-distillation may preserve both knowledge and recall.Latent knowledge can sometimes be elicited through reasoning.Source: §6 · p. 10

Continual learning in LLMs

  • Continual-learning methods address the stability-plasticity tension through regularization, replay, architectural isolation, and distillation, while this work preserves pretrained facts as the stability objective.Unlike SDFT (Shenfeld et al., 2026), which uses on-policy reverse-KL distillation from an in-context teacher to teach new behavior, this method uses an unconditioned frozen snapshot with off-policy forward-KL distillation as a temporal anchor for prior knowledge.Source: §6 · p. 10

7 Conclusion

The paper reframes SFT-induced hallucinations as factual forgetting from representational interference, showing that selective freezing and self-distillation mitigate this trade-off while preserving task or factual learning.

7 Conclusion

  • Self-distillation reduces forgetting from ∼15 to 2 pp without sacrificing plasticity, while selective freezing preserves task learning when new fact acquisition is unnecessary.Both mitigations also transfer to realistic instruction tuning.Source: §7 · p. 10
  • Forgetting scales with representational overlap between new and existing entities, whereas random UUID facts cause near-zero forgetting even at 106 new facts.These findings implicate localized interference rather than capacity or behavioral effects.Source: §7 · p. 10
  • Selective freezing and self-distillation target representational interference, establishing factual stability as a fine-tuning objective rather than an inevitable cost of factual learning.Representational drift analysis suggests self-distillation suppresses interference at the output level, while an associative-memory account predicts forgetting from overlap between new and stored keys.Source: §7 · p. 10

Reproducibility Statement

The paper specifies experimental configurations, implementation details, and supporting appendices, with code and data-construction scripts publicly available. It also discloses LLM assistance in writing and experiments, while attributing research decisions and verification to the authors.

Reproducibility Statement

  • Experimental configurations, protocols, hyperparameters, appendices, and released code and data-construction scripts are specified for reproducibility.The paper reports SLiCK evaluation, training, self-distillation defaults, synthetic-entity construction, Alpaca, representation-geometry analyses, and baseline-sweep details.Source: Reproducibility Statement · p. 11
  • LLMs assisted with prose, boilerplate, refactoring, and documentation, while also serving as experimental models, annotators, and judges; authors verified reported claims.LLAMA-3.3-70B-Instruct was used for Alpaca factuality annotation and AlpacaEval judging.Source: Reproducibility Statement · p. 11

A Results Over Other SLiCK Classification Groups

Across SLiCK knowledge groups, fine-tuning effects depend strongly on prior factual robustness: regular SFT most severely forgets MaybeKnown facts, while Only Known reverses the ordering for WeaklyKnown facts. Self-distillation generally mitigates forgetting but offers little advantage for poorly accessible knowledge.

A Results Over Other SLiCK Classification Groups

  • Experiments use HighlyKnown facts for training and validation to separate QA-format learning from factual learning and obtain a clean forgetting signal.The four selected relations each exceed 30% HighlyKnown prevalence; across n=3,530 development examples, HighlyKnown facts comprise 46.6%.Source: A Results Over Other SLiCK… · pp. 15–16, Table 3 · p. 16
  • On HighlyKnown facts, Only Known preserves ≈95% accuracy, Self-Distillation stabilizes at ≈92%, and Regular SFT degrades to ≈80%.Regular SFT drops roughly 13 pp from the pre-fine-tuning baseline.Source: A Results Over Other SLiCK… · p. 16, Figure 7 · p. 16
  • MaybeKnown accuracy falls from approximately 0.64 to 0.43 under Regular SFT, while Self-Distillation stabilizes at approximately 0.58 and Only Known at ≈0.60.Regular SFT forgets roughly 21 pp, compared with 13 pp on HighlyKnown facts; Self-Distillation recovers approximately 15 pp over Regular SFT.Source: A Results Over Other SLiCK… · p. 16, Figure 7 · p. 16
  • For WeaklyKnown facts, Only Known drops accuracy to approximately 0.04–0.05, below Regular SFT at ≈0.14 and Self-Distillation at ≈0.13.These facts have low baseline accuracy of ≈0.15–0.17; Self-Distillation and Regular SFT converge similarly because the knowledge was not reliably accessible before fine-tuning.Source: A Results Over Other SLiCK… · pp. 16–17, Figure 7 · p. 16

B Results Across Different Models

Across Llama-3.1-8B and Qwen2.5-7B, self-distillation reliably preserves held-out factual knowledge while reducing forgetting relative to regular SFT. Fine-grained freezing further shows that restricting updates within attention is necessary, and partially known facts benefit most from self-distillation.

Cross-model validation

  • The training dynamics replicate across two additional model families and remain consistent in magnitude with the primary 1.5B-parameter experiments.Both architectures show rapid task acquisition followed by progressive forgetting under regular SFT, while Only Known preserves DHeld throughout training.Source: B Results Across Different Models · p. 17

Cross-model training dynamics

  • Self-distillation reduces held-out forgetting from 11–12 pp under regular SFT to 2–3 pp across Llama-3.1-8B and Qwen2.5-7B, mitigating approximately 80%.Self-distillation stabilizes DHeld at approximately 95%, close to the Only Known condition’s 95–97%, while regular SFT reaches approximately 85–86%.Source: Figure 8 · p. 17, B Results Across Different Models · p. 17

Fine-grained freezing

  • Freezing the full attention block is insufficient; stability returns only when updates are restricted to a single projection such as k, v, or o.Jointly updating attention parameters preserves enough factual plasticity to cause forgetting, whereas single-projection updates suppress it sufficiently to protect DHeld.Source: B Results Across Different Models · p. 17

MaybeKnown facts

  • Self-distillation mitigates approximately 90% of MaybeKnown forgetting, stabilizing accuracy at approximately 62–65% versus an 18–19 pp regular-SFT drop.MaybeKnown facts are more vulnerable than DHeld because their partial, weaker representations are more susceptible to interference during new factual learning.Source: Figure 9 · p. 18, B Results Across Different Models · p. 18

C Results Over Different Module Freezing Experiments

Fine-grained freezing experiments show that forgetting is governed by reduced factual plasticity (DUnk), not by which parameter module is restricted. Attention projections and the full attention block suppress plasticity most effectively, while full FFN or full-model updates recover standard SFT forgetting.

C Results Over Different Module Freezing Experiments

  • Freezing controls forgetting by limiting factual integration, while self-distillation offers another control mechanism that preserves stability without sacrificing plasticity.The experiments characterize freezing as one way to reduce DUnk rather than as a uniquely effective structural constraint.Source: C Results Over Different Module… · pp. 18–19
  • DHeld decreases monotonically with DUnk across parameter subsets, showing that factual plasticity—not the specific frozen module—mediates forgetting.Individual attention projections achieve DUnk ≈0.005–0.006, while the full attention block reaches DUnk = 0.010.Source: C Results Over Different Module… · pp. 18–19, Table 4 · p. 19
  • Within the FFN, gate+up drives more plasticity than down, with DUnk = 0.242 versus 0.086, consistent with factual writing into the residual stream.The cited interpretation attributes this difference to the gate+up pathway’s role in writing factual content into the residual stream.Source: C Results Over Different Module… · p. 19
  • Updating the full FFN or full model restores standard SFT dynamics, producing high plasticity and a 15-percentage-point drop in DHeld.The full FFN and full model recover the baseline forgetting pattern, whereas Known provides the stability upper bound.Source: C Results Over Different Module… · p. 19, Table 4 · p. 19

D Results Over Different Parameters for Self-distillation … H Representational Geometry of Synthetic Keys

Across controlled and real-data experiments, self-distillation best preserves pre-existing factual knowledge while retaining acquisition, whereas parameter-space regularization offers weaker protection. Synthetic-key and representation analyses attribute forgetting primarily to pre-existing semantic overlap in entity representations.

D Results Over Different Parameters for Self-distillation

  • Freezing the teacher at snapshot i = 1 gives the highest DHeld, while i = 0 slows DUnk acquisition and later snapshots monotonically reduce held-out stability.The i = 0 result may reflect difficulty learning the QA format rather than factual forgetting alone.Source: D Results Over Different Parameters… · pp. 19–20
  • The distillation weight exposes a stability–plasticity tradeoff: λ = 0.1 and λ = 0.5 accelerate DUnk acquisition, whereas λ = 1.5 reduces forgetting but delays learning; λ = 1 is selected.λ = 1 balances efficient factual acquisition with meaningful forgetting mitigation without an extended training budget.Source: D Results Over Different Parameters… · p. 20
  • Lower temperatures improve DHeld while preserving comparable final DUnk accuracy, so τ = 0.5 is the default; the sweep’s best point is (i = 2, λ = 1, τ = 1).The default configuration retains DHeld = 0.908 at epoch 50, about 2 pp below the 0.93 peak.Source: D Results Over Different Parameters… · p. 20

E Parameter-Space Baselines: Protocols and Sweeps

  • Parameter-space methods reach only 0.78–0.84 final DHeld, at most ∼5pp above standard SFT versus ∼13pp for self-distillation.Table 5 uses the default self-distillation configuration (i = 1, λ = 1, τ = 0.5) at epoch 50.Source: Table 5 · p. 20, E Parameter-Space Baselines: Protocols… · p. 21
  • At near-full acquisition, EWC reaches final DHeld = 0.826 at DUnk = 0.948, LoRA 0.814–0.834 at DUnk ≈ 0.95, weight decay 0.804 at 0.909, and snapshot-anchored ℓ2 0.809 at 0.949.EWC uses an empirical Fisher proxy because pretraining data and trajectories are unavailable; LoRA forgets somewhat less than full fine-tuning on a slower trajectory.Source: E Parameter-Space Baselines: Protocols… · pp. 20–21

F Real-Data Validation on Alpaca: Protocol and Trajectories

  • On Alpaca, factual degradation was 8 pp for SFT, 2 pp for self-distillation, and 1 pp for attention-only freezing, while all exceeded the base model’s 39.8% AlpacaEval rate.At epoch 4, AlpacaEval preferred rates were 50.5% (SFT), 52.3% (self-distillation), and 47.2% (attention-only); differences of roughly ±1–2pp warrant caution.Source: F Real-Data Validation on Alpaca:… · p. 21

G Synthetic Entities Creation Procedure

  • Semantic keys caused 38–41 pp forgetting across value types, whereas UUID keys caused approximately 4 pp, despite all conditions achieving DUnk = 100% and DKnown ≥90%.This dissociates key semantics from answer values as the dominant interference driver.Source: G Synthetic Entities Creation Procedure · p. 23, Table 6 · p. 23

H Representational Geometry of Synthetic Keys

  • Cosine similarity and city-probe scores both order keys from UUID through noisy and token-mix variants, showing graded pre-fine-tuning overlap that tracks forgetting.Last-layer cosine values were 0.45, 0.58, 0.64, and 0.71 for UUID, 2-noise, 1-noise, and token-mix; layer-27 probe scores were 0.29, 0.50, 0.62, and 0.71 respectively.Source: H Representational Geometry of… · pp. 23–24

I Further Analyses on Self-Distillation … J.5 Extension to distributed storage

Further analyses examine self-distillation’s anti-forgetting effects, the overlap-dependent mechanisms underlying interference, and their extension from single to distributed associative-memory storage across transformer components.

I Further Analyses on Self-Distillation

  • Top-k distillation matches full self-distillation, reducing DHeld forgetting to ≈3 pp while DUnk acquisition remains comparable; random-k leaves forgetting at ≈10 pp, like standard SFT.The result indicates that constraining high-probability competing entity tokens, rather than merely constraining the same number of logits, drives forgetting reduction.Source: I Further Analyses on Self-Distillation · pp. 24–25
  • Across six drift metrics, self-distillation consistently reduces hidden-state, neighborhood, targeted, ranking, token-overlap, and distributional drift while preserving comparable DUnk acquisition and better DHeld accuracy than SFT.SFT reaches ≈11% RD versus ≈5%, ≈0.07 NS versus ≈0.02, and ≈−0.09 MTD versus near 0 under self-distillation; Rank-ρ rises from ≈0.32 to ≈0.40 under SFT but stays near ≈0.32 with self-distillation.Source: I Further Analyses on Self-Distillation · pp. 25–27, Figure 13 · p. 26

J Derivations for the Hebbian Associative-Memory Account

  • The Hebbian associative-memory derivations formalize forgetting as cumulative perturbation of held-out readout scores caused by SFT injections and analyze its bias, variance, and dependence on overlap.The appendix introduces notation linking held-out subjects to stored answers and keys, then derives the perturbation δy(A) from cumulative updates.Source: J Derivations for the Hebbian… · p. 27

J.1 Hebbian update derivation

  • Each SFT step produces a rank-one Hebbian increment to the readout matrix, so M new facts yield a first-order sum of such increments when the key map is fixed.The approximation is first order in the learning rate and treats the key map as unchanged during fine-tuning.Source: §J.1 · pp. 27–28

J.2 Bias and variance of δy(A)

  • Under the stated cross-prior regularity assumptions, the systematic pull on a held-out fact is linear in overlap, while variance is bounded through a Gram-matrix analysis.The bias is bounded by η (1 + CR) maxj |ρjA|, and Gershgorin’s theorem yields the variance bound with C ≤2CR + 3.Source: §J.2 · pp. 28, 30

J.3 Forgetting probability and scaling law (heuristic)

  • The heuristic scaling analysis models forgetting as a Gaussian tail whose probability decreases monotonically with semantic-overlap aggregate SA, while output-space penalties suppress overlap-sensitive interference more directly than weight penalties.The Gaussian approximation is used only for monotonic orderings, not to infer a specific exponent; KL regularization targets the affected output directions, whereas weight penalties protect facts uniformly.Source: §J.3 · p. 30, §J.4 · p. 31

J.5 Extension to distributed storage

  • The single-memory analysis extends to factual storage distributed across multiple attention and MLP components, with no qualitative change.Different components may read the residual stream at different layers.Source: §J.5 · p. 31
  • The bias-variance decomposition applies term by term across components, using independence in u as a simplifying assumption.Cross-component covariances share the same u_a terms and are bounded by Cauchy-Schwarz.Source: §J.5 · p. 31
  • The monotonicity law in summed squared overlap is preserved even without independence, with overlaps measured at each component’s layer.The component weights w_c and per-component read layers remain unspecified because they depend on which component dominates retrieval; only the qualitative law is fixed.Source: §J.5 · p. 31

Related papers

Loading 2604.15574v2…