PaperlayerView exact revision on arXiv

Source-linked AI summary

Tracing and Reversing Edits in LLMs

Paul Youssef, Zhixue Zhao, Christin Seifert, Jörg Schlötterer

arXiv:2505.20819v2cs.CL

TL;DR

Knowledge editing can update inaccurate or outdated facts, but it can also be exploited to implant misinformation or bias, motivating countermeasures that do not depend on a list of candidate facts. The paper introduces weight-only approaches to infer edited objects and reverse edits. It reports up to 99% tracing accuracy and 94% reversal accuracy, and concludes that tracing and restoring outputs are feasible without access to the editing operation.

  • Problem

    Prior approaches to detecting altered facts or identifying edit types assume a set of potentially edited facts is available, a condition the paper describes as highly impractical.

    Source: §1 · p. 1

  • Method

    The paper introduces EditScope to infer edited objects from edited weights and uses bottom-rank approximations to reverse edits without information about the edit.

    Source: §1 · pp. 1–2

  • Results

    Up to 99% tracing accuracy and 94% reversal accuracy are reported across several LLMs and knowledge-editing methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results show the feasibility of tracing edits and restoring original outputs without access to the original weights or editing operation.

    Source: §7 · p. 10

    Reversal relies on the assumption that edited objects are not over-represented in bottom-rank approximations of the edited matrix.

    Source: §5 · p. 5

Abstract

from arXiv · show

Knowledge editing methods (KEs) are a cost-effective way to update the factual content of large language models (LLMs), but they pose a dual-use risk. While KEs are beneficial for updating outdated or incorrect information, they can be exploited maliciously to implant misinformation or bias. In order to defend against these types of malicious manipulation, we need robust techniques that can reliably detect, interpret, and mitigate malicious edits. To that end, we introduce the tasks of tracing and reversing edits. We propose a novel method to infer the edited object entity, solely based on the modified weights, without access to the editing prompt or any other semantically similar prompts, with up to 99% accuracy. Further, we propose an effective and training-free method for reversing edits. Our method reverses up to 94% of the edits, and helps regain the original model's output distribution without access to any information about the edit. This method can further be repurposed to distinguish between edited and unedited weights. Our findings highlight the feasibility of tracing and reversing edits based on the edited weights, opening a new research direction for safeguarding LLMs against adversarial manipulations.

1 INTRODUCTION

Knowledge editing can update inaccurate or outdated facts, but it can also be used to inject misinformation or bias. This work proposes weight-only methods to trace and reverse edits, with results reported for both tasks.

1 INTRODUCTION

  • Knowledge editing can update inaccurate or outdated facts at low computational cost, but it can also implant backdoors, misinformation, or bias.The paper presents countermeasures that include retrieving edited or original objects, identifying edited layers, and predicting edited relations.Source: §1 · p. 1, Figure 1 · p. 2
  • More than 88% accuracy across multiple models and more than 85% on out-of-distribution data are reported for inferring edited objects.EditScope derives the edited object from edited weights without access to editing prompts.Source: §1 · p. 1
  • Up to 94% accuracy is reported for reversing edits to retrieve models’ original outputs.The method uses bottom-rank approximations and requires no information about the edit.Source: §1 · p. 2
  • The paper formalizes tracing and reversing edits using model weights alone, without access to the editing prompt or original weights.Tracing targets the edited object, while reversal targets the original output.Source: §1 · p. 2
  • Evaluation across several LLMs and editing methods reports up to 99% accuracy for tracing and 94% for reversal, alongside a new, more challenging editing dataset.The introduction also reports strong generalization on that dataset.Source: §1 · p. 2

2 PROBLEM STATEMENT

The tasks concern identifying an edited fact and undoing an edit, given edited weights but no access to the original edited subset or the editing operation. The stated focus is on countermeasures with minimal assumptions.

2 PROBLEM STATEMENT

  • The setting provides edited weights and unaffected original weights, but neither the pre-edit weights nor information about the editing operation.The paper states that its analysis relies solely on edited weights and does not access the original weights or editing prompt.Source: §2 · p. 2
  • Tracing edits targets the edited object o′, the output an attacker may seek to steer the model toward.The paper also reports results for identifying the relation r in Appendix D.Source: §2 · p. 2
  • Reversing edits intervenes on edited weights so a prompt containing the subject and relation produces the original object o instead of o′.Source: §2 · p. 2

3 MODELS, KES AND DATASETS

The experiments cover four language models, several knowledge-editing methods, and two datasets. The main-body setup uses single edits on selected CounterFact relations, while YAGO supplies a second dataset with more diverse relations.

3 MODELS, KES AND DATASETS

  • Experiments use GPT2-XL, GPT-J, LLAMA3, and QWEN2.5, including representatives of recent LLMs.Source: §3 · p. 3
  • The main body targets rank-one edits such as ROME and r-ROME; appendices test generalization to MEND, MEMIT, and AlphaEdit.Source: §3 · p. 3
  • CounterFact is filtered to 31 of its 34 relations, with 100 successful single edits retained per relation for experiments.The paper also reports reversal generalization to batch and sequential edits in appendices.Source: §3 · p. 3
  • A second dataset samples subject–object pairs from YAGO 4.5 across 15 manually selected relations, excluding relations with fewer than 1,000 pairs.Editing and paraphrased prompts are generated for each relation.Source: §3 · p. 3

4 TRACING EDITS

Section 4 asks whether the edited object can be inferred from edited weights alone. EditScope trains unedited parameters to recover that object without the editing prompt.

4 TRACING EDITS

  • The section frames tracing as identifying the edited object from edited weights without access to the editing prompt.The inferred object can be combined with relation information to reconstruct the edited fact.Source: §4 · p. 3, §4.2 · p. 4

4.1 EditScope

  • EditScope swaps each edited matrix for the absent original and trains unedited model parameters to map a fixed input to its corresponding edited object.The fixed input serves as a placeholder for conventional token inputs; the edited matrix-object pairs serve as training input-output pairs.Source: §4.1 · pp. 3–4, Figure 2 · p. 3

4.2 EXPERIMENTAL SETUP AND RESULTS

  • >88% in-domain and >85% OOD accuracy were achieved across models with ROME and r-ROME when training only the edited layer.The table reports edited-object accuracy for this setting.Source: Table 1 · p. 4, §4.2 · p. 4
  • 99% accuracy for GPT models and >97% for LLAMA3 on CounterFact was achieved when training up to the layer containing the edited matrix.OOD accuracy was slightly lower for GPT-J (-2 p.p.) and LLAMA3 (-3 p.p.); Yago performance dropped slightly because its objects are longer.Source: §4.2 · p. 4, Figure 3 · p. 5

5 REVERSING EDITS

Bottom-rank approximations exclude the largest singular-value components of edited weights to remove edits and recover original outputs. Across models, this approach improves reversal accuracy, recovers output distributions, and can help distinguish edited weights.

5.1 SINGULAR VALUE DECOMPOSITION AND BOTTOM-RANK APPROXIMATIONS

  • Bottom-rank approximations remove the top k singular-value components, aiming to exclude editing information while retaining lower-rank components that may encode the original object.The method uses the edited matrix's SVD and requires no information about the edit.Source: §5.1 · p. 5, §5 · p. 5, §5.3 · p. 6

5.2 ANALYSIS OF RANK-ONE APPROXIMATIONS

  • For GPT2-XL, cosine similarity was 0.98 at k = 1; for GPT-J, it was 0.77 at k = 1 and 0.45 at k = 2, while LLAMA3 rose from 0.20 at k = 1 across k = 2, 3, 4.Similarity drops at larger k for GPT2-XL and GPT-J; QWEN follows GPT-J's pattern with lower values.Source: Figure 4 · p. 5, §5.2 · p. 6

5.3 REVERSAL

  • Reversal accuracy peaked at 94% (GPT2-XL, k = 11), 80% (GPT-J, k = 14), 80% (LLAMA3, k = 15), and 62% (QWEN2.5, k = 13).As k increases, reversal accuracy rises and editing accuracy falls, but the extent varies by model; the method removes edits more effectively than it recovers original objects.Source: Table 2 · p. 6, §5.3 · pp. 6–7

5.2 ANALYSIS OF RANK-ONE APPROXIMATIONS

  • The update-matrix similarity pattern differs by model: GPT models align most at k = 1, whereas LLAMA3's similarity increases across k = 2, 3, 4 before declining.The analysis uses maximum absolute cosine similarity between rows of the update matrix and rank-one approximations.Source: §5.2 · pp. 5–6

5.3 REVERSAL

  • Examples with bottom-rank approximations can differ from original outputs while remaining semantically close.The examples use the best k for each model with ROME and are sampled from a random set of facts.Source: §5.3 · p. 7
  • GPT-J's KL divergence fell from 11.567 to 0.218, while QWEN2.5's fell from 8.988 to 1.534, indicating recovery of the original output distribution.The reported ROME results show a decrease across all models; r-ROME shows a similar pattern.Source: Table 3 · p. 7, §5.3 · p. 7
  • On six GLUE tasks, LLAMA3 models reversed with k = 15 performed on par with edited models and had lower standard deviation.The experiment used 310 ROME edits sampled from 31 CounterFact relations.Source: Figure 5 · p. 8, §5.3 · p. 8
  • Edited weights produced more unique predictions under bottom-rank approximations than unedited weights, suggesting this behavior can distinguish the two.With GPT-J, the average was 1.37 unique predictions for unedited weights versus 2.46 for edited weights; with LLAMA3, it was 1.36 versus 1.84.Source: Figure 6 · p. 8, §5.3 · p. 8

META-LLAMA-3-8B

The section presents examples comparing original outputs with outputs after reversal. In the reported examples, the reversed outputs are semantically close to the unedited outputs.

META-LLAMA-3-8B

  • The displayed examples include cases where the output wording differs after reversal.The examples include fragments such as “soccer” and “footballer.”Source: META-LLAMA-3-8B · p. 9, Table 4 · p. 9
  • The examples compare model outputs after bottom-rank approximations with original outputs, using the best k for each model with ROME.The examples are a random selection of facts.Source: Table 4 · p. 9
  • Outputs after reversal are described as semantically close to the unedited outputs.The examples need not be identical to the original outputs.Source: Table 4 · p. 9

6 RELATED WORK

Related work distinguishes knowledge editing approaches by whether they modify model parameters and describes risks from malicious editing. Existing countermeasures and reversal work address narrower settings than the weight-only tracing and reversal tasks proposed here.

6 RELATED WORK

  • Knowledge editing methods include parameter-modifying approaches and parameter-preserving methods based on memory modules or in-context abilities.Parameter-modifying methods include meta-learning and locate-and-edit approaches.Source: §6 · p. 9
  • Knowledge editing has been used to implant backdoors, spread misinformation, introduce bias, and jailbreak LLMs.The cited examples include Li et al. (2024), Ju et al. (2024), Chen et al. (2024), and Hazra et al. (2024).Source: §6 · p. 9
  • Prior countermeasures classify edited facts or edit types but assume potentially edited facts are available; prior reversal work addresses in-context edits.This paper formalizes tracing and reversal using only model weights.Source: §6 · p. 10

7 CONCLUSION

The conclusion presents weight-only tracing and reversal as feasible without the original weights or edit-operation information. The supplied background describes Transformer processing and ROME’s rank-one updates as context for analyzing edited models.

7 CONCLUSION

  • Weight-only methods infer edited objects, reverse edits to restore original output distributions, and distinguish edited from unedited weights without original weights or edit-operation information.The conclusion presents these tasks as feasible with high accuracy and encourages study in more realistic settings.Source: §7 · p. 10

A AUTOREGRESSIVE TRANSFORMERS

  • Transformer layers process token representations through self-attention and MLP blocks, and the final token’s last-layer hidden state determines the output.The MLP uses a nonlinear function and projection matrices W_K and W_V.Source: A AUTOREGRESSIVE TRANSFORMERS · p. 14

B RANK-ONE MODEL EDITING (ROME)

  • ROME edits factual associations by adding a rank-one update to an MLP projection matrix in a middle layer.The update matrix W_N factors into a column vector and a row vector.Source: B RANK-ONE MODEL EDITING (ROME) · p. 14

C IDENTIFYING EDITED MODELS

C.1 RANK-ONE UPDATE ANALYSIS

The analysis finds that ROME update directions differ across model families and that pairwise cosine similarity rises sharply after editing in GPT models. The authors propose this rise as a cue for distinguishing edited matrices.

C.1 RANK-ONE UPDATE ANALYSIS

  • More than 80% of GPT update-matrix rows share a direction, while LLAMA3 is roughly 50/50; this may shift most GPT rows in one direction.The figure encodes rows with the same direction in blue and opposite directions in orange.Source: §C.1 · p. 14, Figure 7 · p. 15

C.2 ROW VECTOR SIMILARITIES

  • More than 175× with GPT2-XL and relation P190, and more than 25× with GPT-J and relation P138; high edited-matrix similarity helps distinguish edited GPT layers.The figure compares average pairwise cosine similarity across layers for edited and unedited matrices.Source: Figure 9 · p. 15, §C.2 · p. 15
  • Because similarly directed updates may increase row similarity, the authors compare pairwise cosine similarity in original and edited matrices.They measure the relative change; positive values indicate increased similarity.Source: §C.2 · p. 15, Figure 8 · p. 15

D PREDICTING EDITED RELATIONS

Edited matrices contain information that can support relation prediction and edited-object tracing across some knowledge-editing methods. Performance varies by model and method, with strong tracing results for MEND and MEMIT on QWEN and weaker results for other settings.

D PREDICTING EDITED RELATIONS

  • Above 90% accuracy for GPT models and above 75% for LLAMA3 with 2, 3, and 5 relations; edited-matrix relation prediction exceeds a random baseline across class counts.The classifier uses low-dimensional representations of edited matrices.Source: Table 5 · p. 16, D PREDICTING EDITED RELATIONS · p. 16
  • The relation-prediction experiment uses PCA to reduce edited matrices to 50 dimensions, then logistic regression trained and tested on separate samples per relation.The study averages accuracy and standard deviation across five random relation selections.Source: D PREDICTING EDITED RELATIONS · p. 16

E BEYOND RANK-ONE MODEL EDITS

  • The generalization study covers MEMIT and AlphaEdit, both locate-and-edit methods, and MEND, a meta-learning method, but restricts models and data to manage computational costs.The experiments use three random seeds, with model choices constrained by available hyperparameters and compute.Source: E BEYOND RANK-ONE MODEL EDITS · p. 16, §E.1 · p. 16

E.1 TRACING EDITS

  • MEND tracing exceeds 99% accuracy with negligible OOD performance drop, while MEMIT on QWEN is comparable to ROME and r-ROME.The tested methods edit multiple matrices, and tracing trains only the layer preceding those edited layers.Source: §E.1 · p. 17
  • MISTRAL tracing reaches 66% accuracy, while AlphaEdit tracing performs poorly; the authors attribute the latter to weaker edited-object presence after AlphaEdit avoids overfitting.The authors note that hyperparameter tuning might improve MISTRAL performance.Source: §E.1 · p. 17

E.2 REVERSING EDITS

Bottom-rank approximations reverse edits across several methods and settings, but reversal accuracy varies by method and model. The reported batch and sequential outcomes broadly track the corresponding single-edit results for AlphaEdit and across sequential and batch settings.

E.2 REVERSING EDITS

  • AlphaEdit reversal peaks at 81% for GPT2-XL at k = 475 and 64% for LLAMA3 at k = 2162; the authors say larger k values are needed.They attribute this to AlphaEdit projecting changes into the null space of preserved knowledge, making edits less pronounced in top-rank approximations.Source: §E.2 · p. 17

F REVERSING BATCH EDITS

E.2 REVERSING EDITS

  • MEMIT reversal reaches at most 55% on QWEN2.5 and more than 74% on MISTRAL, indicating method- and model-dependent reversal difficulty.The table reports reversal and editing accuracy using bottom-rank approximations.Source: §E.2 · p. 17, Table 7 · p. 19

G REVERSING SEQUENTIAL EDITS

  • With sequential edits, MEMIT peaks at 81% for GPT2-XL, 59% for GPT-J, and 49% for QWEN2.5; AlphaEdit peaks at 81% for GPT2-XL and 62% for LLAMA3.The authors report sequential-setting performance corresponding to batch-setting performance; the figure shows MEMIT results.Source: G REVERSING SEQUENTIAL EDITS · p. 20, Figure 13 · p. 21

E.2 REVERSING EDITS

  • MEND reversal exceeds 70% at k = 1 on both models, while editing accuracy is almost zero and recovering all original outputs remains challenging.The authors suggest edits are mostly localized in the top-1 approximation.Source: §E.2 · pp. 17–18, Table 10 · p. 22

H ADDITIONAL RESULTS

  • The appendix reports additional detailed results and a newly constructed editing dataset for evaluating generalization.It also lists supplementary tables on KL-divergence, relations, update-vector similarities, and edited-matrix dimensions.Source: H ADDITIONAL RESULTS · p. 22

H.1 YAGO DATASET

The YAGO experiments use a dataset built from diverse relations and examine edited-matrix signals and bottom-rank approximations. The reported results describe recovery of original outputs and trends in reversal accuracy as approximation rank changes.

H.1 YAGO DATASET

  • YAGO 4.5 supplies more diverse relations than CounterFact; the experiments use 15 manually selected relations with at least 1,000 subject/object pairs each.DeepSeek R1 generated editing and paraphrased prompts, which were manually checked.Source: §H.1 · pp. 22, 23, 25
  • Training the edited layer or previous layers yields high accuracy for predicting the edited object from the edited matrix.Figure 16 describes accuracy when training different layers.Source: Figure 16 · p. 24
  • More than 80% of update-matrix row vectors have the same direction in the GPT models on YAGO.The section also describes comparisons of edited and unedited matrices across layers and increased row-wise cosine similarity after editing in GPT models.Source: Figure 15 · p. 23, Figure 17 · p. 24, Figure 18 · p. 25
  • Bottom-rank approximations recover the original model’s output distribution in the YAGO experiments.The table caption also describes reversal and editing accuracy for r-ROME, but the supplied passages give no numeric values.Source: Table 12 · p. 23, §H.1 · p. 26
  • The YAGO results include accuracy for predicting the edited relation from low-dimensional representations of edited matrices using logistic regression.Table 18 varies the number of relations, but the supplied caption does not provide accuracy values.Source: §H.1 · p. 25, Table 18 · p. 29

I THE USE OF LLMS

  • As k increases, editing accuracy drops while reversal accuracy increases.The supplied figure and table captions identify reversal and editing accuracy results for bottom-rank approximations, without giving their values.Source: I THE USE OF LLMS · p. 29, Table 19 · p. 29
  • Examples from YAGO show approximate outputs that are semantically close to the original, unedited outputs.The examples use the best k for each model.Source: Table 20 · p. 30
Loading 2505.20819v2…