PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforced Lifelong Editing for Language Models

Zherui Li, Houcheng Jiang, Hao Chen, Baolong Bi, Zhenhong Zhou, Fei Sun, Junfeng Fang, Xiang Wang

arXiv:2502.05759v4cs.CL

TL;DR

Outdated or erroneous knowledge and changing LLM parameters make lifelong editing challenging for hypernetwork-based methods. RLEdit uses reinforcement learning to adapt hypernetwork-generated updates across knowledge sequences, and reports outperforming existing approaches across most evaluation metrics with lower computation time.

  • Problem

    Hypernetwork-based lifelong editing struggles to generalize as LLM parameters change, limiting its effectiveness over long editing sequences.

    Source: §1 · p. 1

  • Method

    RLEdit models the hypernetwork as an RL agent, parameter updates as actions, and LLM performance as reward, optimizing updates across knowledge sequences.

    Source: §1 · p. 2, §3.1.1 · p. 4

  • Results

    RLEdit outperforms existing parameter-modifying methods across most evaluation metrics while requiring only 2% of their average computation time.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RLEdit provides a hypernetwork-based approach to lifelong editing with reported gains in effectiveness, efficiency, and general-capability preservation.

    Source: §6 · p. 9

    Evaluation focuses primarily on factual knowledge modifications in conventional datasets, and post-edited LLM performance on multi-hop information remains unexplored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) acquire information from pre-training corpora, but their stored knowledge can become inaccurate or outdated over time. Model editing addresses this challenge by modifying model parameters without retraining, and prevalent approaches leverage hypernetworks to generate these parameter updates. However, they face significant challenges in lifelong editing due to their incompatibility with LLM parameters that dynamically change during the editing process. To address this, we observed that hypernetwork-based lifelong editing aligns with reinforcement learning modeling and proposed RLEdit, an RL-based editing method. By treating editing losses as rewards and optimizing hypernetwork parameters at the full knowledge sequence level, we enable it to precisely capture LLM changes and generate appropriate parameter updates. Our extensive empirical evaluation across several LLMs demonstrates that RLEdit outperforms existing methods in lifelong editing with superior effectiveness and efficiency, achieving a 59.24% improvement while requiring only 2.11% of the time compared to most approaches. Our code is available at: https://github.com/zhrli324/RLEdit.

1. Introduction

Lifelong editing addresses outdated or erroneous knowledge in LLMs, but hypernetwork editors struggle as model parameters change. RLEdit frames editing as reinforcement learning and reports stronger lifelong-editing results with lower computation time.

1. Introduction

  • Lifelong editing sequentially updates LLM parameters to address stored knowledge that has become outdated or erroneous.The introduction describes it as updating models without compromising their core performance.Source: §1 · p. 1
  • Conventional editors calculate a separate parameter update for each knowledge sample, while hypernetworks learn to map knowledge to updates more efficiently.Repeated matrix operations make conventional methods computationally expensive and error-prone.Source: §1 · p. 1
  • Hypernetwork-based methods struggle when editing changes the LLM: they require training pairs from an identically parameterized model and tend to fail around 100 edits.The introduction contrasts this with locate-then-edit methods supporting up to 5,000 edits.Source: §1 · p. 1
  • RLEdit treats the hypernetwork as an RL agent, parameter updates as actions, and LLM performance as reward, adapting updates to the current model state.Its reward incorporates the difference between the current update and cumulative previous updates to accelerate training and reduce interference between successive updates.Source: §1 · p. 2
  • Across most metrics, RLEdit outperforms existing parameter-modifying methods, uses 2% of their average computation time, and remains satisfactory beyond 20,000 edits.The reported computation time includes hypernetwork training overhead.Source: §1 · p. 2, Figure 1 · p. 2

2. Preliminary

The preliminaries define lifelong editing as sequential parameter updates to one LLM, describe hypernetworks that generate updates from fine-tuning gradients, and review reinforcement learning as an MDP.

2.1. Lifelong Model Editing

  • Lifelong editing sequentially modifies one LLM across knowledge samples, with the goal of matching target outputs while retaining recent and prior knowledge and comprehensive performance.The total number of edits may reach thousands or tens of thousands.Source: §2.1 · p. 2

2.2. Hypernetwork-based Editing Methods

  • Hypernetwork editors map fine-tuning gradients to parameter updates, using rank-1 gradient factors to reduce the burden of d × d weight matrices.The resulting pseudo-activation and pseudoincrement form the update; the paper uses Δ and the update notation equivalently.Source: §2.2 · pp. 2–3

2.3. Reinforcement Learning

  • An MDP specifies states, actions, rewards, a policy, and a discount factor; RL optimizes the policy to maximize discounted reward over a trajectory.At each time step, the agent selects an action from its policy based on the current state.Source: §2.3 · p. 3

3. Method

RLEdit formulates lifelong hypernetwork editing as an RL problem and trains the hypernetwork across knowledge sequences. Its design combines editing objectives with memory backtracking and update regularization, then applies the trained hypernetwork to evolving LLM parameters.

3.1.1. MDP FORMULATION

  • RLEdit models hypernetwork training as an MDP, with the hypernetwork policy generating parameter updates from the current LLM state and knowledge, and the editing loss supplying reward.The process has the Markov property because each step depends on the current LLM parameters and new knowledge, not earlier states.Source: §3.1.1 · pp. 3–4, Table 1 · p. 3

3.1.2. DESIGN OF THE REWARD FUNCTION

  • The basic reward combines target-knowledge updating and unrelated-knowledge preservation losses, balanced by λloc.The losses respectively measure target updating and unrelated-knowledge preservation.Source: §3.1.2 · p. 4
  • Memory backtracking adds losses for the previous k knowledge samples, with exponential temporal decay; its generalization improvement is identified as a key factor in effectiveness beyond 10,000 edits.The component reviews earlier edits alongside the current knowledge to address interference between successive edits.Source: §3.1.2 · pp. 4–5
  • The reward also penalizes the magnitude of generated updates; ablations report that this regularization preserves general capabilities and supports hypernetwork training stability.The regularizer uses the ℓ2 norm of the generated parameter update.Source: §3.1.2 · p. 5

3.2. Training Process of RLEdit

  • 100-fold efficiency improvement over current lifelong editing methods is attributed primarily to optimizing collectively after collecting rewards across the dataset trajectory.The offline approach avoids expensive per-step policy optimization and potential overfitting to specific knowledge samples; γ = 1 weights sequence entries uniformly.Source: §3.2 · p. 5

3.3. Applying RLEdit for Lifelong Editing

  • For lifelong editing, the trained hypernetwork maps gradients from parameter-frozen fine-tuning to LLM updates; the approach can use a 4-layer MLP and plug into most existing hypernetwork-based methods.The paper proposes replacing the basic term in a method’s reward function while adopting its hypernetwork architecture and inputs.Source: §3.3 · pp. 5–6

4. Experiments

Experiments compare RLEdit with editing baselines on three LLMs and datasets, assessing knowledge editing, efficiency, general capability, and portability. RLEdit performs strongly across these evaluations, including as edit sequences grow and when integrated with existing hypernetworks.

4.1. Experimental Settings

  • Experiments use three 8B-scale autoregressive LLMs, three knowledge-editing datasets, and Efficacy, Generalization, and Specificity metrics.The LLMs are Llama-3-8B, Gemma-2-9B, and Mistral-7B-v0.3; datasets are ZsRE, FEVER, and CounterFact.Source: §4.1 · p. 6

4.2. Performance on Knowledge Update and Preservation

  • RLEdit improves Efficacy by 66%, Generalization by 65%, and Specificity by 40% on average over baselines, using only 4% of their time.Conventional single-step methods fall below 50% on Efficacy and Generalization in lifelong editing.Source: §4.2 · p. 7

4.3. Editing with Varying Numbers of Knowledge

  • RLEdit maintains stable performance as edit volume grows and is reported as the first method to scale lifelong editing beyond 10,000 knowledge samples.Baselines degrade progressively, with catastrophic forgetting beyond some thresholds.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4. Editing Efficiency

  • RLEdit takes 0.145 seconds per sample for training and requires 1.14% of the time needed by locate-then-edit methods.The efficiency comparison uses three LLMs on ZsRE under a 20×100 configuration; Figure 4 compares per-sample editing time.Source: §4.4 · pp. 7–8, Figure 4 · p. 7

4.5. General Capability Tests

  • After 3,000 knowledge edits, RLEdit-edited LLMs perform comparably to their pretrained versions across the six GLUE tests.The authors attribute this stability to regularization.Source: §4.5 · p. 8, Figure 5 · p. 8

4.6. Improving MEND/MALMEN

  • Integrating RLEdit’s training methodology improves MEND by 22.89% and MALMEN by 51.86% on average across metrics.The experiments compare the methods across three LLMs and three datasets.Source: Figure 6 · p. 8, §4.6 · p. 8

5. Related Works

Related work covers parameter-modifying and parameter-preserving editing, alongside methods designed for sequential knowledge updates.

Parameter-modifying Editing

  • Parameter-modifying work includes locate-then-edit methods such as ROME and MEMIT, as well as hypernetwork approaches such as MEND and MALMEN.Source: §5 · pp. 8–9

Parameter-preserving Editing

  • Parameter-preserving approaches use external modules or additional parameters, including SERAC’s memory, T-Patcher’s added neurons, and IKE’s retrieval-based examples [Zheng et al., 2023].Source: §5 · p. 9

Lifelong Model Editing

  • Sequential editing methods include dynamic codebooks in GRACE, regularization in RECT, and null-space projection in AlphaEdit.Other approaches use correction neurons, knowledge sharding, singular-value constraints, orthogonalized update directions, or neuron sorting with weight rewinding.Source: §5 · p. 9

6. Conclusion

The paper presents RLEdit as a hypernetwork-based lifelong editing method that formulates editing as reinforcement learning and uses offline updates to retain knowledge sequences.

  • RLEdit formulates lifelong editing as an RL task and uses offline updates to improve retention across entire knowledge sequences.Source: §6 · p. 9
  • RLEdit adds memory backtracking to review prior edits and regularization to mitigate forgetting over long sequences.Source: §6 · p. 9
  • Across several LLMs and datasets, experiments report that RLEdit outperforms baselines in editing effectiveness, efficiency, and general-capability preservation.Source: §6 · p. 9

Limitations

The evaluation has limits in data domain and tested capabilities.

  • The evaluation follows conventional model-editing datasets and focuses primarily on factual knowledge modifications rather than other data domains.Source: Limitations · p. 9
  • The study does not explore post-edited LLMs’ processing of multi-hop information.Source: Limitations · p. 9

Impact Statement

This material describes the experiments’ baselines, datasets, and evaluation metrics, and urges oversight because direct parameter modification can introduce false or harmful information.

Impact Statement

  • Direct parameter modification can introduce false or harmful information, so the authors urge strict validation and oversight for responsible use.Source: Impact Statement · p. 9

A. Detailed Experimental Setup

A.1. Baseline Methods

  • Baselines span hypernetwork, locate-then-edit, and sequential-editing methods, with implementations generally following their original configurations.Source: §A.1 · p. 13

A.2. Datasets

  • ZsRE, CounterFact, and FEVER test editing, paraphrase generalization, and retention of unrelated knowledge.Their dataset structures include target knowledge and equivalent or unrelated examples; CounterFact targets contradictory facts.Source: §A.2 · p. 14

A.3.1. ZSRE METRICS

  • ZsRE efficacy tests target-answer success, generalization tests equivalent queries, and specificity tests retention of unrelated knowledge.Source: §A.3.1 · pp. 14–15

A.3.2. COUNTERFACT METRICS

  • CounterFact efficacy and generalization compare target-answer probability against the original answer, while specificity compares the original answer against the edited output on unrelated inputs.Source: §A.3.2 · p. 15

A.3.3. FEVER METRICS

  • FEVER efficacy and generalization measure target-label matches on edited and equivalent claims, while specificity measures retention of original labels on unrelated claims.Source: §A.3.3 · p. 15

A.4. GLUE Benchmark

The experiments use six GLUE metrics to assess retained general language capability and report RLEdit hyperparameter choices, including sensitivity to the layer setting.

A.4. GLUE Benchmark

  • Six GLUE metrics assess how well editing methods maintain general language capabilities, including SST sentiment, MMLU domain knowledge, and MRPC paraphrase similarity.Source: §A.4 · p. 16

A.5. Hyperparameter Configuration

  • RLEdit uses µ=0.95, k=10, η=1e-4, γ=1, and learning rates of 1e-6 and 1e-5; performance is notably sensitive to the layer setting.The selected settings were based on empirical performance; rank and locality-loss coefficient had limited sensitivity.Source: §A.5 · p. 16

B.1. Ablation Study

The additional experiments examine component contributions, long-sequence editing, general-capability retention, and configuration trade-offs; the appendix also gives the editing algorithm’s input and output.

B.1. Ablation Study

  • Ablations show that RL training is crucial for lifelong adaptation, memory backtracking supports long-sequence efficacy and generalization, and regularization helps preserve general capabilities.Source: §B.1 · p. 17, Table 4 · p. 17

B.2. More Results of Editing with Varying Numbers of Knowledge

  • Most baselines lose effectiveness beyond 5,000 samples, while RLEdit maintains relatively stable editing performance across knowledge-sequence lengths.The comparison covers Llama-3-8B tasks from 20×100 through 150×100; Table 5 reports results under different numbers of edited knowledge samples.Source: §B.2 · p. 17, Table 5 · p. 18

B.3. General Capability Test on CounterFact

  • After 3,000 CounterFact edits, RLEdit preserves general capabilities comparable to the pretrained model, whereas most other methods show significant degradation.Figure 7 presents the general-capability test on CounterFact.Source: §B.3 · p. 17, Figure 7 · p. 19

B.4. Impact of Different Edit Frequencies and Batch Sizes on RLEdit

  • Across seven configurations for 5,000 ZsRE edits, very large batches consume substantial GPU memory, while very high edit frequencies can destabilize training and prevent convergence.The authors recommend choosing edit frequency and batch size appropriately; Figure 8 shows metrics across the configurations.Source: §B.4 · pp. 17, 19, Figure 8 · p. 20

C. Algorithms

  • Algorithm 2 takes a knowledge sequence, pretrained LLM, trained hypernetwork, and hyperparameters, and outputs the post-edited LLM.Source: C. Algorithms · p. 19

1.0 (a) SST (b) MMLU (c) MRPC

  • The displayed Figure 8 reports final metric scores for 5,000 edits under seven configurations, without specifying a winning configuration in its caption.Source: Figure 8 · p. 20

D. Detailed Proof

The appendix formalizes lifelong hypernetwork editing as a Markov decision process and discusses layer choices, few-shot trajectories, and future scope.

D. Detailed Proof

  • The detailed proof models editing as an MDP with parameter-state transitions, controllable update actions, and a bounded, stable reward derived from losses.The state is represented by model parameters and knowledge, or specifically the fine-tuning gradient; Gaussian noise yields stationary transitions.Source: D. Detailed Proof · p. 20

E. More Discussion

  • RLEdit achieves strong results with relatively few trajectory samples, which the authors interpret as suggesting lifelong editing may be simpler than classic RL tasks.Source: E. More Discussion · p. 20
  • Unlike locate-then-edit methods that typically select early layers, RLEdit performs best when editing middle and final layers.Source: E. More Discussion · p. 20
  • Future work includes applying RL to locate-and-edit and memory-based methods, addressing multi-hop and related knowledge, and investigating hallucination and security issues.Source: E. More Discussion · p. 21

F. Dataset Visualization through Examples

The section illustrates lifelong editing with examples from the ZsRE, FEVER, and CounterFact datasets.

F. Dataset Visualization through Examples

Loading 2502.05759v4…