Source-linked AI summary
Popular Knowledge Propagates More Errors in LLM Knowledge Updating
Yuji Zhang, Weibing Wang, Cheng Qian, Duo Zhou, Dilek Hakkani-Tür, Kathleen McKeown, Chengxiang Zhai, Heng Ji
TL;DR
The paper asks which already-correct facts are most vulnerable to collateral corruption during knowledge updates. It constructs FACTPROP and evaluates controlled factual updates, finding that structural popularity predicts both vulnerability and broader error propagation, while PopAnchor uses popular facts for targeted preservation.
Problem
Existing work shows that long-tail knowledge is difficult to acquire and retain, but it is less understood which already-correct facts are vulnerable to collateral corruption during other updates.
Method
The paper constructs FACTPROP from verified factual triples linked by shared entities, measures correct-to-incorrect changes after factual updates, and proposes PopAnchor to preserve popular facts.
Results
Among initially correct facts, those associated with structurally popular knowledge are more likely to be corrupted, while updates involving such knowledge propagate errors more broadly.
Takeaways & Limitations
Structural popularity can guide which existing facts receive preservation priority, and PopAnchor reduces collateral forgetting while outperforming popularity-agnostic and similarity-based baselines.
Takeaways & Limitations
The popularity measure captures object-entity connectivity rather than complete-proposition frequency, while the main analysis uses controlled object substitutions and LoRA fine-tuning.
Abstract
from arXiv · showhide
Updating a language model's knowledge through fine-tuning is essential for keeping its outputs current, yet can also induce factual forgetting and new hallucinations. Prior work shows that long-tail knowledge is harder to acquire and newly memorized long-tail facts are difficult to retain during later fine-tuning. We study a complementary question: among facts that a model has encoded correctly, which are most vulnerable to collateral corruption during other updates? To investigate this question under a realistic factual distribution, we construct a large-scale graph FACTPROP of verified Wikipedia facts by linking triples that share head or tail entities, thereby preserving connections among factual knowledge. We fine-tune models on factual statements and measure correct-to-incorrect facts after each update. Our results reveal a pattern distinct from prior findings on long-tail vulnerability during acquisition and retention: among facts that models already answer correctly, those associated with highly connected entities are more likely to be corrupted by neighboring updates, and updates to such facts propagate errors more broadly. Structural popularity therefore predicts both vulnerability and downstream damage. Inspired by this finding, we propose Popularity-based Anchoring (PopAnchor), a lightweight rehearsal strategy that preserves a small set of popular facts and reduces forgetting.
1 Introduction
Fine-tuning can damage factual knowledge that models previously answered correctly. This paper studies whether structural popularity predicts collateral corruption and uses that signal to target preservation.
- Fine-tuning updates can damage previously correct facts, causing factual forgetting and new hallucinations.
- Prior work emphasizes long-tail knowledge acquisition and retention, leaving vulnerability among already-correct facts less understood.
- FACTPROP links factual triples sharing head or tail entities, then measures correct-to-incorrect changes after selected factual updates.
- Facts associated with structurally popular knowledge are more likely to be corrupted by neighboring updates and produce broader error propagation across graph distances.
- Structural popularity predicts both which updates cause greater downstream damage and which unchanged facts become collateral victims.
- PopAnchor preserves a small set of structurally popular factual prompts during updating and reduces collateral forgetting relative to popularity-agnostic and similarity-based alternatives.
2 Related Work
Related work studies unintended changes beyond update targets through knowledge editing and catastrophic forgetting. These approaches leave open how unexpected corruption is distributed across broader factual knowledge and which examples should receive preservation priority.
- Knowledge-editing studies typically test whether edits propagate to predefined semantically, logically, or compositionally related facts.
- Predefined ripple-effect evaluations do not reveal which broader-distribution facts are unexpectedly corrupted when relevance is unspecified.
- Catastrophic-forgetting studies measure retention across tasks, datasets, or fact sets after sequential fine-tuning, including hallucination and capability degradation.
- Prior findings show that long-tail knowledge is harder to acquire and retain during later fine-tuning.
- Replay, regularization, and parameter-isolation methods mitigate forgetting, but replay effectiveness depends substantially on which examples are selected.
3 FACTPROP: A Factual Graph for Forgetting Analysis
FACTPROP is a verified factual graph paired with natural-language questions that preserves observed connections among knowledge. The graph defines structural popularity through the connectivity of answer entities.
- FACTPROP represents entities as nodes and verified factual triples as directed edges, connecting facts that share head or tail entities.
- Each verified triple is verbalized as a natural-language QA item while graph structure defines factual connections and distance.
- The evaluation retains subject-relation pairs with a unique or primary expected object and uses identical question wording before and after updates.
- FACTPROP contains 100,015 entity nodes and 432,562 verified factual edges across 39 relation types.
- Popularity is defined by the in-degree of an object entity, counting verified facts that point to the same entity.
- The in-degree measure is positively correlated with Wikipedia frequency and pageviews, supporting it as a graph-based proxy for knowledge popularity.
4 Experimental Setup
The experiments apply controlled factual substitutions to selected target facts, then measure changes in previously correct non-target facts at multiple graph distances. Evaluation also checks injection success and unrelated factual performance.
- Models are fine-tuned on selected target facts, and changes in non-target facts are measured after each update.
- The study evaluates four open-weight instruction-tuned models spanning two model families and multiple scales.
- Each update replaces a verified object with a plausible alternative and expresses the injected fact through natural-language QA supervision.
- An update is successful when the injected object receives the highest score among candidate objects for the target query.
- Flip Rate measures retained facts that become incorrect after updating across hop distances 1 through 5, using only facts answered correctly beforehand.
- A fixed set of 50 unrelated factual questions checks whether observed flips reflect propagation rather than broad model degradation.
5 Experimental Results
FACTPROP shows that factual updates can propagate correct-to-wrong flips across multiple graph hops, with structurally popular facts especially vulnerable as affected neighbors and stronger sources of downstream damage. Surface similarity explains only a small subset of errors, while Popularity-based Anchoring reduces propagation with a small preserved-fact set.
- 5.1 Forgetting Error Propagation Persists over Long Distances: Correct-to-wrong flips remain measurable from d = 1 to d = 5, showing that update-induced errors propagate across connected factual knowledge.
- 5.2 Popular Knowledge Is More Vulnerable to Updates: High-popularity facts consistently show higher Flip Rates than low-popularity facts after neighboring updates across all four evaluated models.These facts are grounded in highly connected answer entities but are more easily overturned after nearby updates.
- 5.2 Popular Knowledge Is More Vulnerable to Updates: High-popularity neighbors are more likely to be corrupted even when the updated source is Rare, making them especially susceptible to collateral effects.
- 5.3 Popular Knowledge Causes Wider Error Propagation: Popular-source updates generally produce stronger downstream error propagation than Rare-source updates, with the clearest and most consistent pattern on Qwen3.5-9B.Other models show noisier hop-level variation, but the overall pattern suggests that popular facts can act as stronger sources of update-induced corruption.
- 5.4 Surface Similarity Does Not Explain Most Error Propagation: String similarity does not explain most observed flips: pairs with similarity below 0.4 account for about 90% of evaluated pairs and nearly 89% of all flips, while overall correlation is r = 0.05.Highly similar pairs have around 63% Flip Rate but comprise less than 1% of evaluated pairs; within-neighborhood correlations remain near zero.
- 5.5 Popularity Anchoring Mitigates Error Propagation: PopAnchor reduces error propagation, achieves the lowest Flip Rate across evaluated models and public benchmarks, and remains effective with as few as N=5 anchors.It outperforms Random, Rare, and SimilarAnchor strategies, indicating that selecting structurally prominent facts matters beyond adding an equally sized anchor set.
6 Conclusion
The paper finds that structurally popular knowledge is more vulnerable to collateral corruption during updates, while updates involving it propagate errors more broadly. It introduces PopAnchor, which uses this signal to prioritize preservation of popular facts.
- Conclusion: FACTPROP traces how factual updates affect connected knowledge in a verified graph built from real-world Wikipedia facts.The graph supports analysis of collateral damage among facts that models initially answer correctly.
- Conclusion: Popular knowledge is more likely to be corrupted by updates and updates involving it propagate errors more broadly.This pattern complements prior findings that rare knowledge is difficult to acquire and retain.
- Conclusion: PopAnchor anchors a small set of popular facts to reduce collateral forgetting.It consistently outperforms popularity-agnostic and similarity-based baselines.
- Conclusion: The paper interprets long-tail and popular-fact damage as potentially arising from different mechanisms.This interpretation remains a hypothesis: long-tail facts may contribute little to the objective, whereas popular facts may interfere through existing associations.
Limitations
The study’s popularity proxy measures entity-level connectivity rather than the frequency of complete propositions, and its controlled LoRA setting may not generalize to other update regimes.
- Limitations: The popularity measure uses object-entity in-degree, not complete proposition frequency.Relation frequency and other properties of the full triple may also contribute to vulnerability.
- Limitations: The main analysis uses controlled object substitutions and LoRA fine-tuning with fixed optimization settings.Future work should test broader objectives, data scales, and optimization procedures.
- Limitations: The study’s findings may not capture full-parameter fine-tuning or longer continual-training streams.These settings are outside the main controlled analysis.
7 Ethical Considerations
The paper highlights risks of misinformation, dataset bias, and imbalanced preservation, while describing graph construction and evaluation choices intended to support factual comparisons. Ethical use therefore requires validation, human oversight, and broader signals beyond structural popularity.
- Ethical Considerations: Knowledge updating can alter non-target facts and potentially introduce or amplify misinformation without adequate validation.Practical applications should verify both update success and collateral effects, especially in high-stakes domains.
- Ethical Considerations: The factual graph may inherit coverage gaps, annotation errors, and societal biases from Wikipedia and Wikidata.These source limitations can affect which knowledge is represented and evaluated.
- Ethical Considerations: Prioritizing popular knowledge could favor well-represented entities while protecting long-tail knowledge less.The paper frames structural popularity as a diagnostic and mitigation signal rather than a universal measure of factual importance.
- Ethical Considerations: FACTPROP connects verified factual triples when their subject or object entities overlap, with paths defining evaluation distances.The graph construction preserves factual connections while using verified triples as directed edges.
- Ethical Considerations: The dataset-generation pipeline proposes candidate triples and questions, verifies candidates against Wikidata, and discards unverified candidates.This filtering step supports the graph’s verified-fact construction.
- Ethical Considerations: Evaluation retains subject–relation pairs with a unique or clearly primary expected object to reduce ambiguity in correctness judgments.Relations admitting multiple equally valid answers are excluded.
- Ethical Considerations: Each factual triple is verbalized into a natural-language question whose expected answer is the object entity.Training and evaluation use these questions rather than serialized triples.
- Ethical Considerations: Using the same question string before and after updating controls for prompt variation in measured prediction changes.Grounded triple connections also avoid relying on question-level surface similarity.
A.4 External Validation of the Popularity Proxy
The study validates graph in-degree as a related but non-interchangeable proxy for factual popularity using Wikipedia frequency and pageviews. Agreement is positive overall and strongest among the most frequent entities, while the signals retain distinct meanings.
- 35,868 entities with non-zero values for all three signals form the external validation set, covering 59.9% of 59,932 QID-resolved nodes.
- In-degree is positively associated with Wikipedia frequency and pageviews across both Pearson and Spearman correlations.Pearson r = 0.413 and Spearman ρ = 0.308 for Wikipedia frequency; Pearson r = 0.260 and Spearman ρ = 0.235 for pageviews.
- The top-10 entities by Wikipedia frequency show strong ranking agreement: 8 also fall in the top in-degree decile.
- Graph in-degree, textual frequency, and pageviews capture different aspects of popularity and should not be treated as interchangeable.In-degree counts verified factual relations, whereas frequency and pageviews reflect corpus occurrence and public attention.
B.1 Update Data Construction
The update data combines targeted factual rewrites with neutral and out-of-domain questions, then evaluates learning and collateral effects using candidate scoring and graph-based neighborhoods. This design separates target acquisition from broader knowledge preservation.
- Each target update is expanded into 150 targeted QA examples from 30 templates and paraphrase augmentation, producing 650 total QA pairs with neutral and out-of-domain examples.The remaining examples comprise 400 neutral factual pairs and 100 out-of-domain pairs.
- Candidate scoring tests whether the injected object receives the highest joint log-probability among comparison objects.The protocol distinguishes injection success from open-ended decoding behavior and retains both all-attempted and successful-update-only views for diagnosis.
- Non-target evaluation uses the same QA prompt before and after updating with greedy decoding and alias-normalized, case-insensitive exact match.
- Neighborhoods are expanded continuously from the update source across hops 1–5, and Flip Rate includes only facts answered correctly by the base model.Up to 30 non-target facts are sampled without replacement at each distance.
- Anchoring comparisons use entity-disjoint benchmark batches and a shared N = 100 anchor budget across popular, rare, similarity, and random strategies.
C.1 Four-Model Pooled Analysis
The pooled analysis finds that surface similarity does not explain the broad flip pattern after controlling for update source and graph distance. Similar pairs are rare, and within-group associations are near zero.
- Surface-similarity correlations with flip status range from 0.026 to 0.072, explaining less than 0.6% of outcome variance in every model.
- Within fixed update-source and hop-distance groups, mean and median similarity correlations are near zero with approximately balanced signs.
- Across 588 valid groups, pairs with Levenshtein similarity at least 0.5 comprise only approximately 3% of the controlled evaluation pool.Their scarcity prevents highly similar pairs from explaining the broad error pattern.
- The earlier five-report paired audit is secondary evidence, with weak mixed correlations from −0.09 to +0.06 and only eight high-similarity examples.
D Additional Popularity-Anchoring Results
Additional analyses evaluate popularity anchoring across update-source strata and examine attention perturbations as a diagnostic. Popular anchoring performs best across source groups, while attention differences are consistent only for immediate neighbors.
- The anchoring comparison uses N = 100 prompts, a shared KL behavior-preservation objective, and separate Popular-, Average-, and Rare-source evaluations.
- Popular Anchoring achieves the lowest average Flip Rate for Popular-, Average-, and Rare-source updates.
- Tables 6–8 report negligible or practically small lexical-similarity associations with flip likelihood, including no systematic signal after fixing source and graph distance.
- Popular-source updates produce larger attention perturbations at d = 1 in all three completed paired audits.
- Later-hop attention comparisons are mixed across models, so attention perturbation is treated as a diagnostic of the immediate-neighbor pattern rather than a universal causal mechanism.