PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Editing at Scale leads to Gradual and Catastrophic Forgetting

Akshat Gupta, Anurag Rao, Gopala Anumanchipalli

arXiv:2401.07453v4cs.CLcs.AIcs.IR

TL;DR

Existing model-editing evaluations often cover one or a few edits, leaving the behavior of methods under repeated edits to the same model less fully assessed. The paper evaluates sequential editing at scale and finds that ROME and MEMIT, the strongest methods among those compared, still undergo gradual and catastrophic forgetting that limits their usefulness and scalability.

  • Problem

    Model-editing methods are usually evaluated over one or a few edits, while practical utility requires making multiple edits to the same model.

    Source: §1 · pp. 1–2

  • Method

    The authors sequentially edit the same models and assess editing proficiency, fact forgetting, and downstream task performance, focusing on ROME and MEMIT.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    ROME and MEMIT perform best among the compared methods at scale, but sequential edits bring gradual forgetting followed by abrupt catastrophic forgetting.

    Source: §6 · pp. 8–9

  • Takeaways & Limitations

    Gradual forgetting makes editing less effective as facts are added, while catastrophic forgetting caps scalability, motivating methods that counteract both.

    Source: §6 · p. 9

    The study examines models with 1.5 billion and 6 billion parameters, and the methods may respond differently as models grow larger.

    Source: §7 · p. 9

Abstract

from arXiv · show

Editing knowledge in large language models is an attractive capability to have which allows us to correct incorrectly learnt facts during pre-training, as well as update the model with an ever-growing list of new facts. While existing model editing techniques have shown promise, they are usually evaluated using metrics for reliability, specificity and generalization over one or few edits. We argue that for model editing to have practical utility, we must be able to make multiple edits to the same model. With this in mind, we evaluate the current model editing methods at scale, focusing on two state of the art methods: ROME and MEMIT. We find that as the model is edited sequentially with multiple facts, it continually forgets previously edited facts and the ability to perform downstream tasks. This forgetting happens in two phases -- an initial gradual but progressive forgetting phase followed by abrupt or catastrophic forgetting phase. Both gradual and catastrophic forgetting limit the usefulness of model editing methods at scale -- the former making model editing less effective as multiple edits are made to the model while the latter caps the scalability of such model editing methods. Our analysis also highlights other key limitations of ROME and MEMIT at scale. With our work, we push for the development and evaluation of model editing methods keeping scalability in mind.

1 Introduction

Model editing can correct or add facts in LLMs without updating the entire model, but prior evaluations often focus on one edit at a time. This paper evaluates sequential edits at scale and finds that ROME and MEMIT are vulnerable to localization problems and two phases of forgetting.

1 Introduction

  • Model editing modifies existing knowledge or injects new facts without updating the entire model, a potential alternative to continually pre-training as facts accumulate.The authors motivate editing both as a way to correct incorrectly memorized facts and to add new ones.Source: §1 · p. 1
  • Editing methods are commonly assessed for reliability, generalization across phrasings, and locality, also called specificity.These dimensions measure accurate recall of the edited fact, recall under different phrasings, and effects on unrelated or neighboring facts, respectively.Source: §1 · p. 1
  • Most prior work evaluates model-editing performance one edit at a time, while multiple-edit evaluations have been limited and can show performance declines starting at 10 edits.The paper frames scalability as a central question because practical use may require updating thousands or hundreds of thousands of facts.Source: §1 · pp. 1–2
  • The study evaluates MEND, ROME, and MEMIT against fine-tuning through sequential edits, measuring editing proficiency, fact forgetting, and downstream task performance.The framework studies how model behavior changes as the number of edits increases.Source: §1 · p. 2
  • ROME and MEMIT outperform MEND and fine-tuning at scale, but their edits are less localized than previously believed.The authors report that new edits consistently bleed into other facts stored in the model.Source: §1 · p. 2
  • Sequential editing can cause gradual forgetting of edited facts and downstream abilities, followed by catastrophic forgetting after a disabling edit.A disabling edit leaves new edits unsuccessful, erases recall of previously edited facts, and prevents downstream task performance.Source: §1 · p. 2

2 Methods, Models and Datasets

The study compares ROME and MEMIT with MEND and fine-tuning on GPT2-XL and GPT-J, using CounterFact for sequential editing experiments. It selects CounterFact because QA-format zsRE edits often did not transfer to text-completion recall.

2 Methods, Models and Datasets

  • ROME and MEMIT localize knowledge with causal tracing before updating selected layers; ROME updates one layer, whereas MEMIT updates multiple layers.MEND generates weight updates with a hypernetwork, and fine-tuning is also used as a baseline.Source: §2 · p. 2
  • The study uses GPT2-XL (1.5B) and GPT-J (6B), with ROME and MEMIT as focal methods and MEND and fine-tuning as baselines.The methods edit base language models rather than chat models.Source: §2 · p. 2
  • After successful QA-format edits on zsRE, models failed to produce the correct answer in text-completion prompts in approximately 70% of scenarios.The authors therefore choose CounterFact for their experiments, reasoning that zsRE QA prompts conflate editing with question answering on base language models.Source: §2 · pp. 2–3
  • CounterFact contains counterfactual statements whose target objects are less likely than the original objects, making it a more difficult editing dataset.Its prompts use text completion rather than the question-answer format of zsRE.Source: §2 · p. 3

3 Scaling ROME

Sequential editing causes ROME to gradually lose previously edited facts and downstream ability before an abrupt collapse. The analysis links that collapse to layer-weight drift and finds that a single ROME edit can disable the model.

3 Scaling ROME

  • ROME is evaluated against MEND and constrained fine-tuning on four random 1,000-example CounterFact sequences, with repeated subjects excluded to avoid knowledge conflicts.The samples are intended to reveal patterns independent of fact order.Source: §3 · p. 3

3.1 Editing Proficiency at Scale

  • ROME maintains nearly 100% efficacy until decline begins between 100 and 1,000 edits across samples, while neighborhood accuracy falls beforehand; MEND cannot reliably edit sequentially.The trends are reported across multiple samples and models.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

3.2 Gradual and Catastrophic Forgetting

  • ROME gradually forgets previously edited facts at a slower rate than FT-C, then abruptly forgets many as editability declines; MEND almost instantly forgets prior edits.The authors describe the abrupt phase as catastrophic forgetting.Source: §3.2 · p. 4

3.3 Downstream Evaluation of Edited Models

  • Editing degrades downstream performance across methods: FT-C declines gradually, MEND suddenly, and ROME gradually then abruptly loses task ability at its inflection point.ROME’s abrupt loss coincides with a sharp rise in forgotten facts and a decline in editability.Source: Figure 2 · p. 5, §3.3 · p. 5

3.4 The Source of Forgetting

  • Repeatedly changing one layer while keeping the others fixed may break their compatibility: normalized layer-weight distance rises, then spikes as ROME breaks down.The authors link gradual distance growth to gradual forgetting and the spike to catastrophic forgetting.Source: Figure 2 · p. 5, §3.4 · p. 6

NORMAL EDITS

  • ROME shows gradual and catastrophic forgetting, whereas FT-C shows only gradual forgetting; forgetting includes general task ability as well as previously edited facts.After ROME’s catastrophic-forgetting point, editing becomes ineffective and the model is almost unusable.Source: NORMAL EDITS · p. 6

3.4.1 Disabling Edits in ROME

  • ROME’s disabling edits cripple the model even when applied first; their normalized layer-weight distance is three orders of magnitude larger than for nondisabling edits.The authors call this a fundamental limitation of ROME and note disabling edits could be a source of adversarial attacks.Source: Table 2 · p. 6, §3.4.1 · p. 7

4 Scaling MEMIT

MEMIT’s sequential edits show gradual and catastrophic forgetting, though it forgets fewer edited facts than ROME. Its effects on downstream performance depend on the model, and layer drift accompanies catastrophic forgetting.

4 Scaling MEMIT

  • MEMIT’s editing efficacy is lower than ROME’s, and its edits continuously affect other facts stored in the model.The comparison summarizes ROME and MEMIT at scale, with MEMIT superior across different parameters except editing efficacy.Source: §4 · pp. 7–8, Table 3 · p. 8
  • MEMIT forgets almost three times fewer previously edited facts than ROME, with catastrophic forgetting beginning after approximately 1400 edits.MEMIT also has a longer gradual-forgetting phase across multiple samples.Source: §4 · p. 7
  • GPT-J maintains downstream-task ability as edits accumulate, whereas GPT2-XL gradually loses performance and paraphrase detection before catastrophic forgetting.This shows that some downstream abilities can be lost before a disabling edit cripples the model.Source: §4 · pp. 7–8
  • MEMIT’s edited-layer distance grows gradually until approximately 1400 edits, then spikes at the catastrophic-forgetting point.Editing the disabling fact first does not cause catastrophic forgetting; the authors conjecture that editing a single layer can create larger instabilities because facts are stored across multiple layers.Source: §4 · p. 8

5 Related Work

Prior work evaluates model-editing methods using reliability, generalization, and locality, while other studies examine ripple effects and logical consistency. This paper focuses on methods that modify model parameters and does not evaluate some other memory-based methods.

5 Related Work

  • The paper focuses on parameter-modifying methods, including hypernetwork-generated updates and direct updates to parts of the model.Gupta et al. (2024) unify these approaches under the preservation-memorization framework and enable batched ROME editing with EMMET.Source: §5 · p. 8
  • Other memory-based model-editing methods, including Mitchell et al. (2022) and Zhong et al. (2023), are not evaluated in this paper.Source: §5 · p. 8
  • Prior evaluations use reliability, generalization, and locality, while Cohen et al. (2023) propose testing ripple effects of edited facts.Li et al. (2023b) add knowledge conflict, reverse-edits, and round-edits to evaluate logical consistency.Source: §5 · p. 8

6 Conclusion

Sequential editing can produce gradual forgetting followed by catastrophic forgetting, alongside drift in edited layers. The authors argue that scalable model editing must preserve existing abilities and be evaluated on downstream tasks and recall of edited facts.

6 Conclusion

  • Sequential edits produce gradual loss of edited facts and downstream abilities, followed by catastrophic loss of coherence, function, and editability.The authors attribute both phases to edited layers drifting from their original weights and becoming incompatible with the rest of the model.Source: §6 · p. 8
  • Evaluating scalability should include downstream-task performance and recall of previously edited facts, in addition to efficacy, specificity, and generalization.The paper frames sequential editing as necessary for practical use.Source: §6 · p. 9
  • Gradual forgetting makes sequential editing less effective, while catastrophic forgetting caps how far these methods can scale.Source: §6 · p. 9

7 Limitations

The study evaluates models with 1.5 billion and 6 billion parameters, but larger models may respond differently. Larger-model tests and several batched-edit settings are left for future work.

7 Limitations

  • The study covers 1.5-billion- and 6-billion-parameter models, so its results do not establish how larger models respond to editing.The authors report consistent behavior across the two studied sizes but note that larger models may respond differently.Source: §7 · p. 9
  • The study leaves larger models, increasing batch sizes, and sequential combinations of edit batches unevaluated.The authors describe these settings as out of scope and future work, while stating they can be evaluated with the presented framework.Source: §7 · p. 9

A.1 zsRE Compatibility

The zsRE evaluation uses question-answer prompts, but successful edits in that format often fail to retrieve the fact in text-completion format. This raises a compatibility concern when using zsRE to assess whether knowledge was edited.

A.1 zsRE Compatibility

  • Facts edited successfully in zsRE format are not retrieved in text-completion format 70% of the time.The passage illustrates this mismatch with two examples where generated completions contradict the edited answers.Source: §A.1 · p. 11
  • For the Martha Neumark example, the edited answer is 1904, while the completion prompt asks when she was born and the generated answer states April 15, 1869, in New York City.Source: §A.1 · p. 11
  • The Herb Pomeroy edit returns Oxford University in a completion, not the edited answer, Harvard University.The question-answer edit and the text-completion prompt elicit different college facts.Source: §A.1 · p. 11

A.2 Model Editing Implementation Details

The experiments use default implementations of FT-C, ROME, MEND, and MEMIT, with constrained fine-tuning as the baseline. The fine-tuning gradient-update norm constraint differs by model.

A.2 Model Editing Implementation Details

  • The experiments use the default FT-C, ROME, MEND, and MEMIT implementations described by the MEMIT authors.The implementations are from the authors’ MEMIT repository.Source: §A.2 · p. 11
  • The fine-tuning constraint is applied alongside the default implementations of FT-C, ROME, MEND, and MEMIT.The passage names the editing methods but does not give further implementation details for FT-C.Source: §A.2 · p. 11
  • Constrained fine-tuning limits the gradient-update norm to 5e-4 for GPT2-XL and 5e-5 for GPT-J.These are the default hyperparameters used by the authors.Source: §A.2 · p. 11

A.3 Downstream Evaluation Details

Downstream evaluation measures edited models on four GLUE tasks, using few-shot prompts and a balanced subset of examples. Models are evaluated approximately every 10 edits, with performance measured by F1.

A.3 Downstream Evaluation Details

  • Downstream performance is evaluated on GLUE sentiment analysis (SST2), paraphrase detection (MRPC), natural language inference, and linguistic acceptability classification.The tasks are evaluated with few-shot prompts because the base models do not produce correct answers without in-context prompts.Source: §A.3 · p. 11
  • The models are evaluated approximately every 10 edits on a balanced subset of 200 examples per task, using F1.The subset limits the added computation required to evaluate hundreds of edits on large models.Source: §A.3 · p. 11
  • Table 4 provides the few-shot prompt template used to measure downstream performance on SST-2.The other listed templates cover MRPC, COLA, and NLI.Source: Table 4 · p. 12, Table 5 · p. 13, Table 6 · p. 14

A.4 Additional Scaling Experiments

Across additional samples and model sizes, ROME and MEMIT show editing, forgetting, downstream-performance, and weight-distance patterns that vary by method and sample. The appendix also gives examples of post-forgetting generations and background on the editing algorithms.

A.4.1 Editing Proficiency

  • MEND’s editing efficacy declines significantly before 100 edits; ROME stays near-perfect until a sample-dependent threshold, while MEMIT declines steeply around 4,000 edits on GPT2-XL and before 1,500 on GPT-J.ROME’s near-perfect efficacy lasts until earlier than 250 edits in sample 3 but as late as 1,000 edits in sample 2; ROME and MEMIT also show a consistent decline in neighborhood score.Source: §A.4.1 · p. 16, Figure 5 · p. 17, Figure 9 · p. 18

A.4.2 Forgetting

  • MEND forgets previous edits before 100 edits in all samples, while ROME shows gradual forgetting before catastrophic forgetting and MEMIT sustains gradual forgetting longer.Catastrophic forgetting varies substantially for ROME; for MEMIT it appears before 1,500 edits on GPT-J and around 4,000 on GPT2-XL, near the decline in editing efficacy.Source: §A.4.2 · p. 19, Figure 13 · p. 20, Figure 17 · p. 21

A.4.3 Downstream Evaluation

  • Downstream task F1 falls to zero for MEND before 100 edits, and for ROME and MEMIT at catastrophic forgetting; task performance can degrade even earlier.ROME on GPT-J sample 2 shows significant downstream decline before catastrophic forgetting.Source: §A.4.3 · p. 22, Figure 25 · p. 24

A.4.4 Source of Forgetting

  • ROME’s weight-distance plots show step changes associated with spikes in forgetfulness, while MEMIT’s cross-layer distance rise corresponds to catastrophic forgetting.MEND shows steep linear growth in normalized distance for GPT2-XL layer 47 and GPT-J layer 27.Source: §A.4.4 · p. 25, Figure 29 · p. 26, Figure 33 · p. 27

A.5 Generation Examples

  • The appendix provides GPT-J generations after ROME’s catastrophic forgetting and reports similar failures for MEMIT across models and samples.Table 8 contains the ROME generation examples.Source: §A.5 · p. 28, Table 8 · p. 29

A.6.1 ROME

  • ROME models an updated fact as a key-value transformation and optimizes a replacement value using prompts where the subject appears at the last token.The paper describes ROME as editing specific model weights after locating stored facts; the replacement value is then used to compute MLP weight updates.Source: §A.6.1 · p. 30

A.6.2 MEMIT

  • MEMIT distributes an edit across multiple layers, with an update designed to spread the burden rather than overload one layer.The paper describes MEMIT as a way to scale the number of reliable edits by distributing their impact across layers.Source: §A.6.2 · pp. 30–31

A.6.3 MEND; A.6.4 FT

  • MEND uses an editor network to generate weight updates from rank-one gradient factors, while the fine-tuning baseline constrains each gradient step.The paper also specifies separate edited layers for MEND and fine-tuning in GPT2-XL and GPT-J.Source: §A.6.3 · p. 31, §A.6.4 · p. 31

Related papers

Loading 2401.07453v4…