PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

Lu Dai, Ziyang Rao, Yili Wang, Hanqing Wang, Hao Liu, Hui Xiong

arXiv:2607.08393v1cs.AIcs.CL

TL;DR

Fine-tuned LLMs can memorize injected facts yet fail to use them in downstream multi-hop reasoning. The paper uses self-patching to diagnose this gap and finds that a fixed heuristic recovers 58–75% of oracle headroom across domains and architectures.

  • Problem

    Fine-tuned LLMs can memorize new facts while failing to reliably use them in downstream reasoning, creating accuracy and temporal gaps between memorization and generalization.

  • Method

    The paper fine-tunes LLMs on novel cross-domain knowledge, evaluates chaining and intersection reasoning, and uses self-patching to map where representations become causally usable.

  • Results

    A fixed non-oracle heuristic recovers 58–75% of oracle headroom in failed generalization cases, with results robust across domains, architectures, and models.

  • Takeaways & Limitations

    The findings support knowledge–circuit misalignment as a routing problem in which memorized knowledge is not integrated into mid-layer reasoning circuits.

  • Takeaways & Limitations

    Oracle self-patching is a diagnostic upper bound, and fully adaptive non-oracle alignment strategies remain undeveloped.

Abstract

from arXiv · show

Fine-tuning LLMs to inject new knowledge faces a critical challenge: LLMs can quickly memorize new facts, yet fail to use them for downstream reasoning tasks. We formalize this failure as the \textit{\textbf{Knowing--Using Gap}}, characterized by an accuracy gap and a temporal lag between memorization and generalization. To understand this phenomenon, we fine-tune LLMs with unseen knowledge and monitor the spatial permeation dynamics of the knowledge internally using a novel intervention technique called self-patching. Self-patching identifies activation locations where relocating representations substantially improves failed generalization cases. These results are consistent with a knowledge-circuit misalignment hypothesis: memorized representations can exist internally but may not be routed to computation-effective layers. To demonstrate the practicality of this diagnostic finding, we design a simple heuristic strategy which recovers 58--75\% of the oracle headroom in generalization failure. Experiments are done cross-domain for the robustness of this finding.

1 Introduction

The section identifies a Knowing–Using Gap in LLM knowledge updating: models can memorize unseen facts yet generalize unreliably and later. It introduces self-patching to trace this failure and motivates a knowledge–circuit misalignment hypothesis.

  • Problem: Fine-tuning can produce a “remembering but not using” failure, with models memorizing new facts while failing to apply them in downstream reasoning.This creates a gap between simple memorization and flexible generalization.
  • Problem: The Knowing–Using Gap comprises an accuracy gap, where generalization trails memorization, and a temporal lag, where generalization emerges later.These are the section’s two defining disparities.
  • Experimental setup: The analysis uses unseen knowledge from two real-world domain knowledge bases and evaluates generalization with chaining and intersection reasoning tasks.The datasets eliminate overlap with pretraining; chaining requires a bridge entity, while intersection requires retrieving and filtering attributes for two entities.
  • Method: Self-patching scans layers and fine-tuning progress by relocating hidden representations between runs and measuring changes in correct-answer probability.The method produces a time-evolving spatial map of knowledge permeation and supports analysis of failed generalization cases.
  • Mechanistic hypothesis: The knowledge–circuit misalignment hypothesis proposes that memorized information can be retrievable yet fail to integrate into computation needed for multi-hop reasoning.Continued fine-tuning can bring usable representations into mid-layer computation, coinciding with generalization in some cases.

2 Related Work

This section situates the paper within mechanistic interpretability, LLM knowledge representation and model editing, and grokking-related learning dynamics. Prior work provides methods for explaining model internals, locating factual knowledge, editing facts, and studying delayed generalization.

  • Mechanistic interpretability: Mechanistic interpretability reverse-engineers neural networks into human-understandable components and seeks causal explanations beyond behavioral analysis.Prior methods include observation-based approaches such as logit lens, linear probes, and sparse auto-encoders.
  • Knowledge Representation in LLM: The Linear Representation Hypothesis and Key-Value Memory framework model factual knowledge as linear activation-space directions, often stored in MLP weights.This localization motivated model-editing methods such as ROME, which directly modify MLP weights to update specific facts.
  • Grokking and Learning dynamics: Grokking describes validation generalization improving suddenly long after training accuracy has saturated, and it has been observed beyond small algorithmic tasks.Recent work reports grokking in pre-training and fine-tuning of large transformers.

3 Dataset preparation

The dataset pairs memorization QA tasks for injecting new facts with disjoint generalization QA tasks that test applying those facts through novel reasoning structures. It uses biomedical and academic STaRK subsets and defines chaining and intersection evaluations over supporting fact triplets.

  • Dataset design: The dataset pairs memorization QA tasks with generalization QA tasks, so models first learn supporting facts and then apply them in downstream evaluation.Memorization tasks provide finetuning material, while generalization tasks are not explicitly memorized.
  • Dataset design: The dataset is adapted from STaRK’s biomedical STaRK-Prime and academic STaRK-MAG subsets to test robustness across domains.STaRK is a heterogeneous real-world knowledge base containing millions of entities and relations.
  • Knowledge representation: Each knowledge atom is a fact triplet f = (n1, e12, n2), representing a relation e12 between head entity n1 and tail entity n2.For example, (MRE11, ppi, ATRX) represents that protein MRE11 interacts with protein ATRX.
  • Task construction: Memorization tasks query a head entity and relation, requiring the model to predict the corresponding tail entity.For example, the query asks what protein interacts with MRE11, with ATRX as the answer.
  • Task construction: Generalization tasks comprise chaining and intersection: chaining requires sequential reasoning over two supporting tasks, whereas intersection identifies shared entities under relation-specific noise.These tasks evaluate chain reasoning and identifying relationally shared entities, respectively.
  • Evaluation separation: Memorization and generalization tasks are disjoint by construction, with single-hop training queries and multi-hop evaluation queries using novel templates.Fact entities may overlap when chaining requires bridge entities, but the compositional query structure remains unseen during training.

4 The Phenomenon: Characterizing the Knowing–Using Gap

The Knowing–Using Gap separates rapid memorization of injected facts from delayed or incomplete downstream use. It is characterized by an end-of-training accuracy gap and a temporal lag in reliable generalization across tasks, fine-tuning methods, and scales.

  • Definitions: Memorization accuracy measures direct recall of injected facts, whereas generalization accuracy measures downstream task performance requiring reasoning with those facts.These are evaluated as time-dependent performance curves, Amem(t) and Agen(t; T).
  • Definitions: The gap has two dimensions: the accuracy difference after convergence and the temporal lag between memorization saturation and reliable generalization.Temporal lag excludes failed cases and uses a stability window of w consecutive epochs.
  • Across-task pattern: Memorization reaches near-ceiling accuracy within a few epochs under chaining, while downstream use remains low for an extended period and varies by task.Under LoRA, memorization saturates quickly across tasks, but downstream use shows delayed emergence and task-dependent ceilings; intersection approaches minimal lag and high final use accuracy.
  • Fine-tuning method comparison: FFT generally memorizes substantially faster than LoRA, but this advantage does not consistently produce faster or better downstream use.For chaining, FFT has a comparable lag and nearly identical final use accuracy; for intersection, it has a larger lag and lower Agen than LoRA.
  • Model and data scale: Increasing model size does not eliminate temporal lag, while increasing injected-fact count tends to widen the final accuracy gap despite strong direct recall.The results indicate that scaling storage does not directly yield proportional gains in reasoning.

5 Mechanistic Analysis for Knowledge–Circuit Misalignment

The section explains the Knowing–Using Gap as knowledge–circuit misalignment: fine-tuning can store answer-relevant representations without routing them through computation-effective layers. Self-patching provides interventional evidence for this account and motivates a heuristic that recovers much of oracle generalization headroom.

  • Knowledge–circuit misalignment hypothesis: Fine-tuning can encode new facts in storage states that support recall but fail to route them into computation-effective layers required for multi-step reasoning.This spatial misalignment is proposed as the mechanism behind generalization failure.
  • Self-patching intervention: Self-patching relocates an anchor representation across source and target layers, measuring the change in target-prompt correctness to identify computation-effective placements.A positive layer-pair effect indicates that the source layer contains a representation useful when placed at the target layer.
  • Knowledge permeation dynamics: Before memorization, patching rarely helps; after memorization, off-diagonal gains appear while diagonal failures persist, and successful cases later expand across the diagonal.The dynamics indicate that knowledge becomes internally available before naturally reaching computation-effective locations.
  • Interventional results: Chaining accuracy lifts by 1.5–6× in every Table 4 cell, while memorization is near-perfect and unassisted downstream use remains low across models and domains.Oracle self-patching also nearly eliminates the smaller know-use gap in intersection.
  • Ablations and practical strategy: Head-entity patching yields the largest improvements, cross-context effects preserve layer-pair structure, and the fixed heuristic recovers 58–75% of oracle headroom.These findings support entity-linked knowledge representations and show that the diagnostic can inform a practical alignment strategy.

6 Conclusion

The paper identifies the Knowing–Using Gap: fine-tuned LLMs can memorize injected facts yet fail to use them in multi-hop reasoning. Self-patching traces this failure to knowledge–circuit misalignment, where knowledge is stored but does not reach reasoning-relevant mid-layer circuits.

  • Fine-tuned LLMs memorize injected facts yet fail to use them in multi-hop reasoning.
  • Self-patching causally diagnoses the gap as knowledge–circuit misalignment.The method traces the failure to the internal routing of knowledge representations.
  • Knowledge is encoded in storage-oriented early or late layers but does not permeate mid-layer circuits required for reasoning.The conclusion characterizes the gap as a routing problem rather than a capacity problem: knowledge resides in the model, but not where reasoning happens.

A Dataset details · A.1 Construction pipeline · A.2 Requirements validation

The dataset is built through an automated, scalable pipeline that generates memorization and generalization QA pairs from knowledge-graph paths. It is validated for scalability, diversity, real-world grounding, cross-domain robustness, and minimal prior model knowledge.

  • A.1 Construction pipeline: The automated pipeline samples knowledge-graph facts and generates QA pairs using LLM-based or template-based methods.The LLM-based approach promotes linguistic diversity, while the template-based approach supports structural consistency.
  • A.1 Construction pipeline: Manually designed meta paths specify valid entity and relation sequences for each generalization task.For example, a chaining path can connect anatomy, gene/protein, and drug entities through specified relations.
  • A.1 Construction pipeline: Sampled paths are decomposed into fact triplets that form memorization tasks, with additional noise facts added for intersection-task complexity.Memorization questions are generated using an AI assistant or predefined templates.
  • A.1 Construction pipeline: Corresponding generalization questions are generated from the same sampled paths using an AI assistant or templates.This aligns each generalization task with the path used to construct its memorization task.
  • A.2 Requirements validation: 8 million distinct fact triplets in STaRK-Prime KG provide the scalability basis, while 3 generalization tasks derive from 53 meta paths spanning 10 entity types and 18 relation types.The dataset’s real-world knowledge is grounded by a biomedical knowledge graph.
  • A.2 Requirements validation: 1,000 sampled facts yielded zero-shot accuracies around or below 0.06 on both STARK-PRIME and STARK-MAG, indicating little prior model knowledge.The evaluation used pretrained LLMs without fine-tuning on the corresponding memorization tasks.

B Experiment Setup Details · C Cross-Domain Replication Details (STaRK-MAG) · C.1 Domain comparison

The experiments separate memorization from downstream use across STaRK-PRIME and STaRK-MAG, using controlled fine-tuning, checkpoint tracking, and self-patching diagnostics. Cross-domain replication preserves the protocol while testing differences between biomedical and academic knowledge graphs.

  • B Experiment Setup Details: STaRK-PRIME is the default domain, while STaRK-MAG provides cross-domain replication; chaining tests held-out compositional queries requiring recovery of a bridge entity.Each chaining instance contains support facts E1 →r1 →E2 and E2 →r2 →E3, followed by E1 →r1 →r2 →?.
  • B Experiment Setup Details: Temporal-lag and accuracy-gap experiments fine-tune open-weight LLaMA and Qwen models while tracking memorization and downstream-use accuracy at every checkpoint.The practical injection setting uses batch size 10, learning rate 10^-4, and 50 epochs, with AdamW weight decay 0.01.
  • B Experiment Setup Details: Multi-fact LoRA runs use rank r = 16, α = 32, dropout 0.05, and attention and MLP projection modules across all transformer blocks.The reproduction configuration uses learning rate 2×10^-4, effective batch size 8, fp32 gradients, and random seed 42.
  • B Experiment Setup Details: Self-patching scans source-target layer pairs at head-entity token positions, recording whether replacing target residual representations with source representations improves downstream answers.The training-dynamics scan uses 100 randomly sampled chaining tasks over 30 epochs.
  • B Experiment Setup Details: The oracle diagnostic reports each instance’s best layer pair, whereas the fixed heuristic uses two predetermined pairs per model without per-instance search.This setup distinguishes an upper-bound diagnostic from a deployable fixed strategy.
  • C Cross-Domain Replication Details (STaRK-MAG): STaRK-MAG follows the STaRK-PRIME protocol but covers academic publications, contrasting biomedical entities through naming conventions, relation types, and graph density.The MAG patching evaluation uses 500 chaining instances and 166 filtered intersection instances.
  • B Experiment Setup Details: Controls vary patched token positions and compare direct generation, chain-of-thought prompting, irrelevant-fact patching, and correct-fact self-patching under exact-match scoring.Aggregate proportions are reported with 95% Wilson confidence intervals.

C.2 Training dynamics

Training dynamics reveal task-dependent Knowing–Using gaps: chaining rapidly memorizes while generation remains near zero, whereas intersection learns both abilities in tandem. This persistent contrast holds across Qwen model scales and mirrors the biomedical-domain pattern.

  • C.2 Training dynamics: Chaining reaches >0.95 memorization by epoch 15 while generation remains near zero throughout training.This is the starkest decoupling observed across the Qwen MAG training runs.
  • C.2 Training dynamics: Intersection shows memorization and generation rising in tandem to similarly high final values.Its near-parallel learning curves are consistent with the small gap observed in both domains.
  • C.2 Training dynamics: The chaining gap remains large and persistent, unlike intersection’s near-parallel learning curves.Figure 7 tracks memorization and generation accuracy over 50 training epochs across tasks and model scales.

D Scaling Results on Knowing–Using Gap

Scaling model size does not eliminate the temporal lag between memorization and generalization, while injecting more facts tends to widen the final accuracy gap despite strong direct recall. On LLaMA-3.2-8B, chaining exhibits a larger delay than intersection, which remains closer to memorization.

  • Scaling Effects: Increasing model size does not eliminate the temporal lag ∆T.The knowing–using gap persists across model scales.
  • Scaling Effects: More injected facts tend to widen the final accuracy gap ∆A(T), even when direct recall remains strong.Scaling storage does not directly translate into proportional gains in usable reasoning.
  • Task Differences: On LLaMA-3.2-8B, chaining shows a larger temporal lag than intersection.Intersection remains much closer to memorization than chaining.

E More results on knowledge-circuit misalignment. … H Limitations and Responsible Research Details

The supplementary results reinforce knowledge-circuit misalignment through architecture-dependent storage locations, controlled self-patching recovery, statistically significant Knowing–Using Gaps, and additional permeation cases. The supplied passages do not report limitations or responsible-research details.

  • E More results on knowledge-circuit misalignment.: PatchScope finds LLaMA’s detectable early-layer knowledge storage location lower than Qwen’s across model scales.This suggests LLaMA tends to store knowledge in early to middle layers.
  • E More results on knowledge-circuit misalignment.: Controlled experiments show self-patching achieves the strongest recovery for both chaining and intersection.Irrelevant patching remains below the knowledge-specific intervention.
  • F Statistical Significance and Confidence Intervals: 95% Wilson score confidence intervals are reported for all main results.Wilson intervals are preferred over normal approximations for proportions near 0 or 1.
  • F Statistical Significance and Confidence Intervals: n = 1000 facts are used for memorization evaluation, while chaining and intersection use n = 500 and n = 1000, respectively.These sample sizes define the reported evaluation settings.
  • F Statistical Significance and Confidence Intervals: p < 10^-50 for every model establishes that the per-instance Knowing–Using Gap is significant under McNemar’s test.The test uses paired 0/1 outcomes, and the gap is defined as Mem.−Chain.
  • G More cases on permeation dynamics: Additional LLaMA-3.1-8B and Qwen-2.5-7B cases show permeation dynamics consistent with the main results.Patch-effective regions covering the diagonal indicate emergence of direct chaining ability.

H.1 Limitations … NeurIPS Paper Checklist

The paper identifies limitations in self-patching’s localization, oracle alignment, predictive capability, and token-layer granularity. It also documents reproducibility resources and compute, released assets and licenses, ethics safeguards, and checklist compliance.

  • H.1 Limitations: Self-patching may understate recoverable headroom because knowledge can distribute across multiple positions or be redundantly encoded beyond the fixed anchor intervention.The method intervenes at one anchor position while moving representations across layers.
  • H.1 Limitations: Oracle best-pair results are diagnostic upper bounds, while fully adaptive non-oracle alignment strategies remain an open direction despite the fixed heuristic’s practical feasibility.The heuristic appears in Section 5.5; the passage does not quantify its recovery.
  • H.1 Limitations: The mechanism repairs misalignment post hoc but lacks an early-training signal predicting which facts will fail to generalize.A predictive metric could enable proactive interventions during fine-tuning.
  • H.1 Limitations: The mechanistic analysis localizes knowledge at the token-layer level rather than to specific attention heads or MLP sublayers.Finer-grained localization could refine the knowledge–circuit misalignment hypothesis and support more targeted interventions.
  • H.2 Reproducibility and Compute: Code, data, and reproduction instructions are anonymously released, with Appendix B specifying data splits, filtering, hyperparameters, evaluation rules, and statistical tests.Experiments used NVIDIA A800 GPUs with 81,920 MiB memory, an Intel 128-core CPU, and 512 GB system memory; typical runs used 8 GPUs and took about 5 hours to 1 day.
  • H.3 Assets, Licenses, and Released Artifacts: The study uses STARK-PRIME and STARK-MAG, open-weight Qwen2.5 and LLaMA models under their stated public licenses, and releases derived QA data and code under the MIT License.The released QA data are generated from knowledge-graph facts and templates or AI-assisted wording, without consent-bearing human-subject data.
  • H.4 Ethics, Broader Impacts, and Safeguards: The authors report conformity with the NeurIPS Code of Ethics and state that the work involves no crowdsourcing, human-subject experiments, participant compensation, or IRB-style review.The paper also discusses positive diagnostic impacts and negative risks from improving adaptation to incorrect, unsafe, or harmful facts.
  • NeurIPS Paper Checklist: The checklist reports that the claims match the paper’s scope, experimental details are specified, statistical analyses are documented, and released assets are documented alongside code and data.Reported statistical materials include 95% Wilson score confidence intervals, McNemar’s tests, exact binomial tests, Spearman correlation analysis, and temporal-lag sensitivity checks.
Loading 2607.08393v1…