Knowledge Editing for Large Language Models: A Survey

Motivation

• Naive fine-tuning can be computationally intensive and may overwrite valuable existing knowledge because it changes pre-trained parameters without constraints [20] [31] [172].Even parameter-efficient fine-tuning strategies may still require substantial computational resources [158] [174].Source: §1 · p. 2

KME

• KME precisely updates specific knowledge without harming irrelevant knowledge, using auxiliary parameters or updates to partial model parameters, sometimes with constraints on unmodified behavior.The survey describes KME as a way to update LLMs without explicit model retraining.Source: §1 · p. 2

KME objectives

• KME must achieve locality by preserving outputs for unrelated inputs and generality by adapting consistently to semantically related inputs.For example, changing the answer about a country’s president should not alter knowledge about another country’s leader, but should affect related queries about the same leader.Source: §1 · pp. 2–3

Related work

• Earlier surveys cover fine-tuning, knowledge-enhanced models, or selected KME methods, but this survey addresses KME’s locality and generality and offers a broader treatment of methods and applications [159] [63] [172] [152].The introduction identifies gaps in prior work’s coverage of KME categorizations, datasets, technical details, and the range of existing methods.Source: §1 · p. 3

Contributions

• The survey formulates KME as constrained optimization and classifies methods as external memorization, global optimization, or local modification, analyzing each category’s methods, strengths, and limitations.It presents the categories as specialized cases of the general formulation, with refined constraints.Source: §1 · p. 3
• The survey examines evaluation metrics, datasets, realistic applications, practical challenges, and future research directions for KME.It also analyzes methods’ effectiveness and feasibility to help practitioners select approaches for particular tasks.Source: §1 · pp. 3–4

2.1 Editing of Machine Learning Models

• Model editing precisely changes pretrained models to encode specific knowledge while preserving existing knowledge and behavior on unrelated inputs [68].Editing may target downstream performance, bias reduction, human-preference alignment, or novel information [18] [20] [103] [39] [44] [72] [88] [177]. Earlier examples include editing GAN layers as linear memory [8], editable training, classifier decision rules, constrained factual fine-tuning [177], and transformer MLP key-value memories [42] [43].Source: §2.1 · p. 4

2.2 Language Models

• Transformers underpin LLMs and typically stack self-attention modules with residual connections alongside point-wise feed-forward network layers.The fully fledged encoder-decoder transformer was initially designed for neural machine translation.Source: §2.2.1 · p. 5

2.2.1 Transformers.

• Multi-head self-attention combines information across tokens, while FFN layers add information stored in fully connected weights.Attention supports next-token prediction from sequence relationships and dependencies; final transformer representations support token- and sequence-level tasks.Source: §2.2.1 · pp. 5–6
• LLMs are transformers with billions of parameters that exhibit emergent factual and commonsense knowledge, and are categorized as encoder-only, encoder-decoder, or decoder-only models.Examples include BERT, T5, and GPT and LLaMA [144], respectively.Source: §2.2.2 · p. 6

2.3 Relevant Topics

• Fine-tuning, unlearning, and RAG overlap with KME but do not fully address its goal of updating generalized knowledge inherent in LLMs.Fine-tuning can be costly and impractical for black-box LLMs [172] [173]; unlearning removes specific training samples rather than abstract knowledge [105], while RAG retrieves documents without updating inherent model knowledge [40].Source: §2.3 · p. 6

Editing Target

• An edit replaces an encoded knowledge triple t = (s, r, o) with t* = (s, r, o*), defining the update as e = (s, r, o → o*).Here, s is the subject, r the relation, and o* the new target object replacing the original object o.Source: §3 · p. 6

Input and Output Space

• A subject-relation pair becomes a natural-language prompt x, and model output y maps back to an object; multiple textual pairs may express the same triple.The formulation calls prompts for an edit’s subject and relation its in-scope input space, with original and target output spaces associated with the old and new objects.Source: §3 · p. 7

Formulation

• KME updates a pretrained model for edits while minimizing changes to input-output pairs irrelevant to those edits.The loss L measures discrepancy between the edited model’s output and a desirable target response; M(f; E) denotes the modification applied to the model.Source: §3 · pp. 6–7
• The objective requires generality on in-scope prompts and locality on unrelated prompts, preserving pretrained outputs across most inputs.Locality matters because knowledge requiring updates is often only a small fraction of the model’s total knowledge.Source: §3 · p. 7

4.1 Accuracy

• Accuracy measures success on predefined edit prompt-answer pairs [17] [29] [106] [174] [175] and enables fair comparisons, but long-output tasks lack consensus test designs [22].It is straightforward to apply to categorical tasks such as fact checking.Source: §4.1 · p. 8

4.2 Locality

• Locality measures agreement between edited and original models on out-of-scope inputs, preserving unrelated knowledge; because the full input space may be infinite, methods predefine feasible test ranges.SERAC generates hard out-of-scope examples by sampling semantically similar training inputs from zsRE.Source: §4.2 · p. 8

4.3 Generality

• Generality measures exact-match correctness on semantically similar in-scope prompts, helping ensure updated knowledge generalizes beyond training inputs and does not overfit a particular prompt.Equivalent inputs may differ in syntax, morphology, genre, or language.Source: §4.3 · p. 9

4.4 Portability

• Portability measures transfer of an edit to logically related edits, commonly involving reversed or neighboring relations, so edited knowledge supports reasoning beyond specific input formats.These related edits can assess robust knowledge transfer [21] [172].Source: §4.4 · p. 9

4.5 Retainability

• Retainability measures performance change after each consecutive edit; unlike scalability, it concerns consistency across edits, with higher retainability indicating less change in overall performance.T-Patcher [66] explored sequential editing, while SLAG found editing performance dropped when multiple beliefs were updated continuously; many existing approaches fall short.Source: §4.5 · pp. 9–10

4.6 Scalability

• Scalability measures whether an editing strategy can incorporate many edits simultaneously; using current techniques sequentially is ineffective when vast numbers of edits are needed.SERAC supports at most 75 edits, while MEMIT enables thousands without significant influence on editing accuracy. The relative measure accounts for single-edit baseline accuracy when assessing performance after multiple edits.Source: §4.6 · p. 10

5 METHODOLOGIES

• The section presents a taxonomy of KME strategies based on how and where new knowledge is injected into pre-trained LLMs.It discusses the advantages and drawbacks of the strategies.Source: §5 · p. 10
• It introduces methods from each KME category and analyzes their technical details and insights.Source: §5 · p. 10
• The method analysis also addresses shortcomings and relationships among the approaches.Source: §5 · p. 10

5.1 Categorization of KME Methods

• External memorization stores edits in extensible external memory without changing pretrained weights, preserving pretrained knowledge and supporting massive edits with controllable memory size.It is suited to massive editing when computational resources are limited.Source: §5.1 · p. 11
• Global optimization incorporates new knowledge through optimization strategies that limit effects on unrelated knowledge, but may be inefficient with many parameters to optimize.It favors generality of edited knowledge, as optimization can promote learning relevant knowledge [2].Source: §5.1 · p. 11
• Local modification locates and updates parameters relevant to an edit, potentially changing only a small fraction and improving memory efficiency over memorization and computational efficiency over global optimization.Source: §5.1 · p. 11
• The taxonomy distinguishes external memorization, global optimization, and local modification according to where and how edits introduce new knowledge into an LLM.The categories differ in their strengths and weaknesses across four evaluation metrics.Source: §5.1 · p. 11

5.2.1 Overview.

• External memorization adds trainable parameters ω to encode new knowledge while retaining the original parameters ϕ, enabling scalable edits with minimal perturbation but raising storage concerns.The strategy’s storage and invocation requirements can become substantial as the amount of edited knowledge grows.Source: §5.2.1 · pp. 11–12, §5.2.4 · p. 14

5.2.2 Memory-based Strategies.

• Memory-based methods store edits outside the model, including SERAC’s cached examples and human-feedback approaches such as Language Patch and MemPrompt.SERAC uses a scope classifier to route matching prompts to a counterfactual model; Language Patch supports creating, editing, or removing natural-language patches, while MemPrompt uses corrective feedback prompts.Source: §5.2.2 · p. 12
• IKE [174] injects factual edits through in-context demonstrations, while MeLLo [175] addresses multi-hop editing by retrieving relevant memory facts for query subquestions.Prompt-based edits can handle factual knowledge but not more complex multi-relation questions; MeLLo decomposes queries into subquestions and retrieves pertinent facts to help answer them.Source: §5.2.2 · p. 13

5.2.3 Extension-based Strategies.

• Extension-based methods add parameters through FFN refinements, adapters, hook layers, or subject-word embedding updates, with examples including CALINET [29], T-Patcher [66], GRACE [52], COMEBA-HK [81], and SWEA [82].T-Patcher adds a limited number of neurons in the final FFN layer, and its patches can be flexibly sized and activated for specific edits; GRACE uses a deferral mechanism to decide when to use its dictionary.Source: §5.2.3 · pp. 13–14

5.3.2 Constrained Fine-tuning.

• Constrained fine-tuning restricts changes to non-target knowledge through loss-change constraints or parameter-norm limits; RECT [48] updates only the top-k% of parameters with the largest numerical updates.Zhu et al. [177] propose an approximate constraint, while RecAdam [13] anneals the balance between fine-tuning loss and parameter distance to limit large early updates.Source: §5.3.2 · p. 15
• Other constraints preserve original knowledge with PPA LoRA plug-ins, MELO clustered LoRA blocks, or F-Learning [106], which subtracts parameters associated with knowledge to forget.PPA uses K-adapter [153] to identify in-scope inputs and route them to plug-ins; MELO assigns separate non-overlapping LoRA blocks to edit clusters.Source: §5.3.2 · pp. 15–16
• Meta-learning [36] [145] minimizes expected loss across edit tasks to reduce overfitting; Editable Training separates base, edit, and locality losses and applies k gradient steps before its objective.Its locality loss is computed on samples outside the edit set.Source: §5.3.2 · p. 16

5.3.3 Intermediate Fine-tuning Strategies.

• Intermediate fine-tuning uses small hyper-networks to generate main-model weight updates; KE [25] predicts constrained edits, while MEND transforms gradients with low-rank auxiliary networks.KGEditor [17] combines a hyper-network with an added FFN-architecture layer to store knowledge and edit knowledge-graph embeddings.Source: §5.3.3 · pp. 16–17

5.3.1 Overview.

• Updating all parameters favors precision, generality, and editing relevant knowledge, but makes locality difficult to preserve and can make fine-tuning costly, unstable, and prone to overfitting.Global optimization methods are mostly model-agnostic, but balancing generality against locality remains important.Source: §5.3.1 · pp. 14–15, §5.3.4 · p. 17

5.4 Local Modification

• Local modification locates weights storing obsolete knowledge and edits them to produce the correct answer, changing only a small fraction of model weights to protect unrelated information.This approach avoids retraining or fine-tuning the whole model.Source: §5.4.1 · pp. 17–18

5.4.1 Overview.

• FFN neurons are treated as knowledge neurons: attention supplies relevant input context, while FFN weights retrieve and inject factual information into the representation.The first-layer weights act like keys and the second-layer weights like values or memory.Source: §5.4.1 · p. 18

5.4.2 Groundtruth-based Strategies.

• KD [22] and DEPN locate knowledge-bearing weights through cumulative changes in the probability of the groundtruth answer, with DEPN targeting privacy knowledge.KD assesses individual FFN weights; weights with high cumulative probability change are treated as relevant to the answer.Source: §5.4.2 · pp. 18–19
• ROME uses a rank-one FFN weight update, while MEMIT extends the approach to mass edits shared across MLP layers in a top-down manner.ROME derives a target activation for the subject and edits the second-layer FFN weights to produce it; MEMIT spreads edit residuals across critical layers.Source: §5.4.2 · p. 19
• PMET changes how MEMIT residuals are distributed, EMMET [50] derives closed-form updates under exact edits, and DINM [149] locally modifies layers identified as storing toxic knowledge.PMET uses a square-root strategy to allocate residuals to bottom FFN layers; DINM uses discrepancies between toxic and non-toxic sequence embeddings and non-toxic samples.Source: §5.4.2 · p. 19

5.4.3 Prompt-based Strategies.

• ROME randomly masks hidden key-vector representations of prompt tokens to identify relevant layers; MEMITCSK [49] analyzes subject, verb, and object corruption for commonsense editing; BIRD [93] targets reversibility.Source: §5.4.3 · p. 20
• Local modification methods still lack scalability and retainability, as performance may deteriorate when more edits are performed.Source: §5.4.4 · p. 20

6 DATASETS

• KME datasets divide into generation tasks with textual outputs and classification tasks with discrete labels, drawing on varied sources adapted for model editing.Classification datasets offer precise evaluation without requiring a specific output space.Source: §6 · p. 20, §6.2 · p. 23

6.1 Generation Datasets

• Generation datasets predominantly use relational facts to test factual edits, with outputs commonly requiring prediction of an object across paraphrased or multi-hop contexts.Relational datasets provide more definitive answers, making evaluation more convenient and precise [172].Source: §6.1 · p. 20
• zsRE, CounterFact, WikiGen, T-REx-100 and T-REx-1000, and ParaRel [32] test paraphrase resilience, generalized factual edits, false-knowledge detection, and relational prompts.zsRE includes human-generated paraphrases and Wikipedia provenance; CounterFact uses Wikidata entities; T-REx variants contain 100 or 1,000 facts; ParaRel covers 34 relations with 253,448 prompts for 27,738 facts.Source: §6.1 · pp. 21–22
• NQ-SituatedQA, MQuAKE [175], and Hallucination [52] evaluate time- or location-dependent answers, multi-hop updates, and correction of false biography continuations.MQuAKE-CF targets counterfactual edits and MQuAKE-T temporal updates; Hallucination provides 1,392 potential test edits pairing GPT-3 biography prompts with Wikipedia continuations.Source: §6.1 · p. 22
• MMEdit [16] supports multimodal LLM editing with VQA and image-captioning tasks; ECBD evaluates propagation of entities absent from pretraining; VLKEB [65] binds images to entities; MLaKE [163] spans five languages.MLaKE reports relatively high generalization for languages within the same language family compared to those from different families.Source: §6.1 · pp. 22–23
• UKE [166] tests updates from unstructured Wikipedia texts, RippleEdits [21] measures edits induced by an existing edit, and Conflict/Round Edit probes knowledge conflicts and distortion.RippleEdits contains 5,000 edits with varied ripple effects; Conflict/Round Edit is built from WikiData using logical rules.Source: §6.1 · p. 23

6.2 Classification Datasets

• Classification datasets assess edits through prompt-target pairs and include fact verification, topic sentiment, occupational associations, evidence-claim entailment, and label-shifted court-topic classification.FEVER excludes claims lacking sufficient information; ConvSent tests changing sentiment about one topic without altering responses to others; Bias in Bios evaluates whether the ground-truth occupation ranks first; VitaminC-FC uses evidence as the edit descriptor; SCOTUS tests 11 topics across periods with shifting label distributions.Source: §6.2 · pp. 23–24

7.1 Question Answering

• KME enables on-the-run QA updates while preserving unrelated knowledge; CuQA tests recurrent edits, and MQuAKE tests multi-hop reasoning, where MeLLo performs outstandingly [175].MQuAKE requires edited models to update answers to implicit relations several hops from the edited fact.Source: §7.1 · p. 25

7.2 Fact Checking

• Fact-checking models need continual updates as claims evolve; prior work edits facts while protecting unrelated performance, and SERAC uses Wikipedia evidence descriptors that generalize to same-page inputs [177].The cited work identifies transformer components conducive to factual edits; SERAC reports significant improvements over baselines.Source: §7.2 · pp. 25–26

7.3 Natural Language Generation

• KME supports NLG through FRUIT’s compressed Wikipedia-article updates and MEND’s more coherent Wikitext continuations, producing richer target text than QA or fact-checking.FRUIT uses T5 and a compressed output format; MEND targets credible 10-token extensions from Wikitext prefixes using multi-layer token-wise activations and gradients.Source: §7.3 · p. 26

8.1 Challenges

• KME faces a locality–generality trade-off: MEMIT and ROME generally preserve higher locality by locating target-knowledge sites, while larger T-Patcher [66] memories increase locality but decrease generality.Theoretical analysis is less developed than practical KME methods; gives Bayesian theoretical support for identifying harmful training examples and erasing their information, while LEACE [9] offers an analytical framework for erasing target-concept information from every layer.Source: §8.1 · p. 26
• Editing at scale remains difficult because many edits can undermine coherence, locality, and generality, while external memories and sequential single-edit methods remain inadequate.Large numbers of edits can contradict existing knowledge; external memorization remains vulnerable at thousands of edits, and sequentially applying single-edit techniques is suboptimal.Source: §8.1 · p. 27
• Current KME methods underperform on unstructured updates, including when knowledge triples are extracted from unstructured texts, motivating more robust methods for such inputs.The UKE benchmark [166] evaluates updates from unstructured texts and reports significant performance declines for state-of-the-art methods.Source: §8.1 · p. 27

8.2 Future Directions

• Optimization-free KME could avoid parameter updates through prompt-based strategies, but existing IKE [174] is difficult to scale and often has unsatisfactory retention.IKE uses demonstration contexts to guide copying, updating, and retaining predictions across prompts.Source: §8.2 · p. 27
• Automating target discovery could reduce the labor and cost of finding knowledge to edit, though crawling, converting, querying, and correcting answers remains inefficient.The proposed direction is to identify, evaluate, and prioritize updates from raw sources such as websites and social media without human effort.Source: §8.2 · pp. 27–28
• Continual editing should rectify emergent issues on the run while preventing edited models from forgetting or contradicting previous edits.This addresses the mismatch between one-step offline editing and applications such as online QA models receiving reports of incorrect answers.Source: §8.2 · p. 28
• Robust KME should preserve accuracy while resisting adversarial perturbations, misinformation, and malicious attempts to inject harmful knowledge.The discussion emphasizes safeguards against unwanted inputs, especially as LLMs are increasingly open-sourced.Source: §8.2 · p. 28
• Editable fairness could use KME to mitigate pre-training biases, but addressing fairness across groups rather than specific people remains challenging.The discussion gives occupation-biography classification as an example where nuanced injected knowledge might support more equitable treatment.Source: §8.2 · p. 28

• The survey formulates KME as a constrained optimization objective balancing editing accuracy and retention, and reviews evaluation metrics that clarify desirable attributes of edited models.Source: §9 · p. 28
• Its structured taxonomy categorizes KME techniques, outlining central challenges and representative methods while discussing their strengths and weaknesses.Source: §9 · p. 28
• The survey summarizes datasets used to assess KME techniques and notes that some methods require specific dataset structures for training or evaluation.Source: §9 · p. 29
• It highlights real-world applications of KME techniques to encourage more practical implementations.Source: §9 · p. 29
• The survey identifies potential future research challenges and offers directions for advancing the field.Source: §9 · p. 29