Massive Editing for Large Language Models via Meta Learning

• Incorrect or outdated knowledge motivates post-training LM editing, but fine-tuning risks overfitting and catastrophic forgetting.Source: §1 · p. 1
• Hyper-networks generate targeted parameter shifts and can explicitly optimize editing generalizability and locality.Source: §1 · pp. 1–2
• Existing hyper-networks scale poorly for synchronous edits because they overfit to the LM’s current state and require costly retraining after updates.Source: §1 · p. 2
• MALMEN aggregates shifts through a least square formulation and separates hyper-network and LM computation to reduce memory use for arbitrary batch sizes.Source: §1 · p. 2
• MALMEN evaluates editing up to thousands of facts across BERT-base, GPT-2, T5-XL, and GPT-J on fact verification and question answering.Source: §1 · p. 2

• Model-editing research spans constrained fine-tuning, external memory, locate-and-edit methods, and meta-learning.Source: §2 · p. 2
• External-memory methods store edits or add narrowly activated neurons so unrelated inputs remain unchanged.Source: §2 · p. 2
• Locate-and-edit methods identify knowledge-bearing neurons or hidden states and modify associated value representations.Source: §2 · pp. 2–3
• Meta-learning methods transform fine-tuning gradients into targeted parameter shifts, using rank-1 masks or low-rank processing of key and value gradients.Source: §2 · p. 3

• Equivalent tuples are semantically equivalent reformulations of an edit, whereas unrelated tuples fall outside its equivalent collection.Source: §3 · p. 3
• The LM maps each prompt-answer tuple to p(y|x) and is parameterized by trainable linear-layer weights W.Source: §3 · p. 3
• The editing objective simultaneously handles m prompt-answer tuples, their equivalent tuples, and unrelated tuples whose predictions should remain unchanged.Source: §3 · p. 3
• Editing performance is measured by editing success, generalization success, and locality success.Source: §3 · p. 3

4.1 AGGREGATING PARAMETER SHIFTS USING THE NORMAL EQUATION

• MALMEN computes each token-conditioned shift from the key and standard fine-tuning value gradient, then forms value differences and updates selected linear layers simultaneously.The procedure can edit a subset of linear layers or tokens, while changes in earlier layers alter later-layer keys as in standard fine-tuning.Source: §4.1 · pp. 4–5
• Existing MEND-style summation can suffer cancellation because shifts for different keys may have opposing magnitudes, whereas MALMEN matches their induced value differences across keys.The method treats Transformer linear layers as key-value memories and aggregates shifts by preserving their effects on the corresponding keys.Source: §4.1 · p. 4
• MALMEN formulates aggregation of hyper-network parameter shifts as a regularized least-squares problem and applies the normal equation to obtain one shift with nearly equivalent effects.For keys U and value differences D, the solution is S* = D U^T (U U^T + λI)^−1; regularization improves numerical stability when U is not row-wise full rank.Source: §4.1 · p. 4

4.2 MEMORY ECONOMIC TRAINING

• Traditional meta-gradient computation caches hyper-network intermediates for all edited facts, causing excessive memory costs when thousands of facts are edited.The number of cached key-gradient tuples scales with edited linear layers and prompt tokens.Source: §4.2 · p. 5
• MALMEN separates LM and hyper-network back-propagation: it accumulates constant-size LM gradients first, then computes per-tuple meta-gradient components through a proxy loss.Equivalent and unrelated tuples are processed in batches, and gradients can be accumulated across batches.Source: §4.2 · pp. 5–6
• This decomposition preserves the computation amount and result of the traditional procedure while substantially reducing memory, allowing thousands of facts to be edited on one GPU.The meta loss combines generalization on equivalent tuples with locality preservation on unrelated tuples; Figure 1 depicts the contrasting cached-intermediate approach.Source: §4.2 · pp. 5–6, Figure 1 · p. 5

5 EMPIRICAL EVALUATION

• The evaluation spans BERT-base, GPT-2, T5-XL, and GPT-J on closed-book fact verification and question answering, with edit counts scaled across model sizes.BERT-base uses FEVER, while GPT-2, T5-XL, and GPT-J use zsRE; the large-model experiments reach 16,384 edits.Source: §5 · pp. 6–7, §5.1 · p. 8

5.1 SCALING CURVE

• MALMEN demonstrates the strongest scalability on BERT-base and GPT-2, editing over an order of magnitude more facts than MEND with similar performance.The small-model scaling curves vary the number of edits from 2^4 to 2^12 for BERT-base and 2^6 to 2^12 for GPT-2.Source: §5.1 · pp. 7–8, Figure 4 · p. 7, Figure 5 · p. 7
• On BERT-base, fine-tuning is fragile and shows a larger ES–GS gap than MEND and MALMEN, whereas hyper-network shifts exhibit better generalization.The fine-tuning failures arise from sensitivity to the learning rate in the described editing setup.Source: §5.1 · p. 8
• MALMEN edits more than two orders of magnitude more facts than MEND on GPT-J with similar performance, while outperforming MEMIT and taking 12.25 hours versus MEMIT’s 33.75 hours for 16,384 edits.The comparison uses GPT-J (6B) and a single NVIDIA A100 GPU, including training time.Source: §5.1 · p. 8

5.2 ABLATION STUDY

• Caching only answer-producing tokens accelerates training and inference by 38.9% on a single NVIDIA A40 GPU, although caching all tokens performs better on BERT-base before larger-scale editing reverses that advantage.The memory-efficient separation allows MALMEN to edit thousands of facts on a single GPU, while MEND-style computation quickly exceeds the hardware limit.Source: Table 1 · p. 8, §5.2 · p. 9, Table 2 · p. 9
• Replacing MALMEN’s normal-equation update with summed parameter shifts produces cancellation that increases with edit count, while MALMEN’s mean residual is about three orders of magnitude smaller.The residual measures the offset caused by parameter-shift aggregation, and the comparison is conducted on GPT-2.Source: §5.2 · p. 9, Figure 8 · p. 9

6 DISCUSSION

• MALMEN aggregates parameter shifts through a least-squares formulation and updates language-model parameters with the normal equation.The approach is designed to support editing thousands of facts under constrained memory.Source: §6 · p. 9
• MALMEN separates hyper-network and language-model computation to permit arbitrary batch sizes under limited memory budgets.Source: §6 · p. 9
• MALMEN still requires computation linear in the number of facts to compute the meta gradient and does not generalize to rephrasings beyond prompts.The authors identify broader consistency under local editing as an unresolved challenge.Source: §6 · p. 9

A.1 DETAIL OF MEND

• MEND generates pseudo keys and pseudo gradients from the key and value gradient, producing a rank-1 parameter shift controlled by a learning-rate parameter.Source: §A.1 · p. 12

A.2 EFFICIENT IMPLEMENTATION OF MEND AND MALMEN

• The efficient implementation avoids explicit computation of all parameter shifts because storing them is spatially demanding.Source: §A.2 · p. 12
• MALMEN forms a value-difference matrix and outputs the optimal parameter shift using the normal equation.Source: §A.2 · p. 13

A.3 KEY SHIFTING ALSO EXISTS IN THE STANDARD FINE-TUNING

• For a single linear-layer key, gradient descent on weights induces a value update equivalent to gradient descent on the value with learning rate ηuT u.Source: §A.3 · p. 13
• When multiple layers are updated, later-layer keys shift after editing, yet training more layers usually improves fine-tuning and MALMEN performance.Source: §A.3 · p. 13

A.4 FINE-TUNING BERT-BASE ON THE FEVER DATASET

• BERT-base is fine-tuned with Adam at learning rate 3e-5, selecting two epochs after validation and then tuning on all data.Source: §A.4 · p. 13

A.5 EDITING MODELS BY FINE-TUNING

• Editing updates the last three Transformer-block FFNs in BERT-base and GPT-2, the last two encoder-decoder FFNs in T5-XL, and one GPT-J FFN layer.Source: §A.5 · p. 13
• The fine-tuning editing baseline uses AdamW with learning rate 5e-4, weight decay 5e-4, and five epochs to report results.The stated training trend is higher ES and GS and lower LS with more epochs.Source: §A.5 · p. 13

A.6 HYPER-PARAMETERS OF MEND AND MALMEN

• MEND and MALMEN share a hyper-network rank of 1920, two blocks, initial learning rate 1e-6, meta-learning rate 1e-5, locality coefficient 1, and maximum meta-gradient norm 1.Source: §A.6 · p. 13

A.7 JUSTIFICATION FOR THE RE-IMPLEMENTATION OF MEND

• The original MEND implementation exceeds the memory budget for editing many facts and uses hyperparameters considered suboptimal for multiple editing.Source: §A.7 · p. 13
• The re-implementation changes the pretrained model, answer selection, unrelated-tuple sampling, metric computation, and Edit Success calculation relative to the original experiment.Source: §A.7 · pp. 13–14
• The revised implementation achieves consistently higher Edit Success and Accuracy Drawdown, which the authors attribute to using locality coefficient λloc = 1 rather than 10.Source: §A.7 · p. 14, Table 3 · p. 14

16 November 1739 ✓

• The Pellicia example contrasts an incorrect answer, “Psperiidae,” with the corrected equivalent-prompt answer, “Hesperiidae.”Source: §16 · p. 14

A.8 QUALITATIVE ANALYSIS ON THE FAILURE CASES OF MALMEN

• MALMEN’s post-edit predictions usually stay close to target answers, with errors generally limited to individual tokens.The examples come from editing T5-XL on zsRE; Tables 4 and 5 cover equivalent and unrelated tuple failures.Source: §16 · p. 14, Table 4 · p. 14, Table 5 · p. 15

16 November 1739 ✓

• When MALMEN errs on an edit tuple, it often makes the same error on the equivalent tuple.Source: §16 · p. 14
• Some cases show correct answers to equivalent prompts even when the corresponding edit prompt is answered incorrectly.Source: §16 · p. 15

B PROOFS

• Theorem 1 derives the gradient of a differentiable function composed with the update mapping D U^T (U U^T + λI)^−1 with respect to D.Source: B PROOFS · p. 15
• Theorem 2 derives the derivative of the composed function with respect to λ, yielding a trace expression with a negative sign and squared inverse matrix.Source: B PROOFS · p. 15