PaperlayerView exact revision on arXiv

Source-linked AI summary

Reverse-Engineering Model Editing on Language Models

Zhiyu Sun, Minrui Luo, Yu Wang, Zhili Chen, Tianxing He

arXiv:2602.10134v2cs.CRcs.AIcs.CL

TL;DR

Locate-then-edit parameter updates can leak the data they are intended to edit. The paper introduces KSTER to recover edited subjects and prompt context, and reports high recovery alongside a decoy-based defense that mitigates leakage while preserving editing utility.

  • Problem

    Whether locate-then-edit parameter changes create a leakage side channel matters because the paradigm is used to modify model parameters without retraining to mitigate memorized sensitive information.

    Source: §1 · p. 1

  • Method

    KSTER uses update-matrix analysis and spectral analysis for subject recovery, then entropy-based prompt recovery; subspace camouflage injects semantic decoys into the update subspace.

    Source: §1 · pp. 1–2

  • Results

    Over 99% subject recall and 88% semantic similarity are reported on Llama3-8B-Instruct with CounterFact, across experiments spanning three models and three editing methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Subspace camouflage is reported to mitigate leakage without compromising editing efficacy.

    Source: §1 · p. 2

    Prompt recovery has failure cases: semantic ambiguity can push the true prompt into ranks 6–100, while confused edits can arise when edit objectives conflict with statistical priors.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Large language models (LLMs) are pretrained on corpora containing trillions of tokens and, therefore, inevitably memorize sensitive information. Locate-then-edit methods, as a mainstream paradigm of model editing, offer a promising solution by modifying model parameters without retraining. However, in this work, we reveal a critical vulnerability of this paradigm: the parameter updates inadvertently serve as a side channel, enabling attackers to recover the edited data. We propose a two-stage reverse-engineering attack named \textit{KSTER} (\textbf{K}ey\textbf{S}paceRecons\textbf{T}ruction-then-\textbf{E}ntropy\textbf{R}eduction) that leverages the low-rank structure of these updates. First, we theoretically show that the row space of the update matrix encodes a ``fingerprint" of the edited subjects, enabling accurate subject recovery via spectral analysis. Second, we introduce an entropy-based prompt recovery attack that reconstructs the semantic context of the edit. Extensive experiments on multiple LLMs demonstrate that our attacks can recover edited data with high success rates. Furthermore, we propose \textit{subspace camouflage}, a defense strategy that obfuscates the update fingerprint with semantic decoys. This approach effectively mitigates reconstruction risks without compromising editing utility. Our code is available at https://github.com/reanatom/EditingAttack.

1. Introduction

Locate-then-edit methods can expose edited information through the parameter changes intended to remove it. The paper analyzes this leakage, evaluates KSTER, and proposes subspace camouflage as a defense.

1. Introduction

  • Locate-then-edit methods modify selected parameters without retraining, but their updates can expose the information they aim to edit.The introduction frames this as an underexplored safety question about leakage through the editing process itself.Source: §1 · p. 1
  • The authors theoretically show that update-matrix row spaces encode edited-subject fingerprints and use spectral analysis to recover subjects.KSTER then applies entropy-based prompt recovery to reconstruct the edit’s semantic context.Source: §1 · pp. 1–2
  • Over 99% subject recall and 88% semantic similarity are reported for Llama3-8B-Instruct on CounterFact.The experiments cover three models and three editing methods.Source: §1 · p. 2
  • Subspace camouflage injects semantic decoys to obscure update fingerprints and mitigate leakage while preserving editing utility.The paper also reports theoretical robustness against a broad class of white-box attacks.Source: §1 · p. 2

2. Related Work

Prior work studies multiple LLM security and privacy risks, while research on privacy risks specific to locate-then-edit remains scarce. The paper distinguishes its update-based recovery from approaches requiring original prompts or recovering less edit information.

2. Related Work

  • Locate-then-edit is a popular, parameter-efficient and interpretable approach, but its privacy risks remain sparsely studied (Meng et al., 2022; 2023; Fang et al., 2025).Related work also situates model editing among LLM security and privacy research, including jailbreaks, backdoors, membership inference, and personal-information leakage.Source: §2 · p. 2
  • Youssef et al. (2025) recover pre-edit behavior but not editing prompts or original outputs; Patil et al. (2024) detect original outputs assuming access to original prompts.The paper contrasts these approaches with recovery from parameter updates without original prompts.Source: §2 · p. 2

3. Preliminaries

The preliminaries describe locate-then-edit as constrained weight updates that remap subject key vectors while preserving general knowledge, then specify the attacker’s access and assumptions.

3.1. Model Editing in LLMs

  • Editing represents facts as subject, prompt-template, and target-output triples, and focuses on ROME and variants that update specific FFNs storing factual associations.The main text studies single-time editing; sequential editing results are left to Appendix F.Source: §3.1 · p. 2
  • The edit seeks an update ΔW that remaps a subject key vector to a target value through a constrained least-squares objective while preserving general knowledge.For single-fact editing, ROME uses a rank-one analytical solution; MEMIT generalizes the approach to batch editing, and AlphaEdit projects updates onto the null space of Kp.Source: §3.1 · pp. 2–3

3.2. Threat Model

  • The white-box attacker sees pre- and post-edit weights, the editing algorithm, and its hyperparameters; the gray-box attacker sees only before-and-after output logits.The study mainly evaluates white-box attacks and includes a gray-box baseline.Source: §3.2 · p. 3
  • Attackers are assumed not to know the edited data, but to have the covariance matrix and a candidate pool containing the true edit.The covariance-access assumption is relaxed in an ablation, and the candidate pool is considered feasible to construct by filtering public trends or domains.Source: §3.2 · p. 3

4. The KSTER Attack

KSTER reverses locate-then-edit updates to recover edited subjects and prompts, exploiting subject-stable activations and update-subspace structure. Evaluations report high recovery across settings while identifying semantic-generalization and optimization-conflict failure cases.

4. The KSTER Attack

  • KSTER uses parameter differences in two stages: infer edited subjects from candidate activations, then recover prompts from candidate relations.The recovered subject and prompt can be used to obtain the original output from the initial model.Source: §4 · p. 3

4.1. Subject Invariance in LM Model Editing

  • Across prompt templates, same-subject hidden states remain highly similar at the shallowest edited layer, allowing candidate scoring with a generic template.Attention focuses more than 50% on subject tokens across templates.Source: Figure 2 · p. 4, §4.1 · p. 4

4.2. Attack Stage I: Subject Inference

  • The update’s row space preserves an edited-subject signature; spectral reconstruction recovers the key-vector subspace for projection-based candidate ranking.The paper also proves exact recovery under stated assumptions and robustness to covariance perturbations for MEMIT.Source: §4.2 · pp. 4–5

4.3. Attack Stage II: Prompt Recovery

  • Prompt recovery ranks candidates by relative entropy reduction, exploiting the sharp uncertainty drop for edited prompts compared with non-target prompts.The attack can then query the initial model for the original output when subject and prompt inference are correct.Source: §4.3 · p. 5

4.4. Attack Evaluation

  • Subject recall stays 0.94–1.00 across batch sizes versus gray-box 0.43–0.55 at N = 100; CounterFact prompt top-20 recall is 0.94–0.99 and output recovery exceeds 70%.The prompt-recovery results also report semantic similarity of 0.78–0.88 on zsRE.Source: Table 1 · p. 6, Table 2 · p. 6, §4.4 · p. 7
  • Attack performance against MEMIT converges at 100 covariance samples across three data sources, while AlphaEdit maintains peak performance with just 10 samples.The sources tested were Wikipedia, WikiText, and Pile.Source: §4.4 · p. 7
  • Ranks 6–100 reflect semantic type ambiguity, while ranks 700–1k arise when edit targets conflict with covariance priors and entropy rises.The resulting tail rankings compromise edit specificity.Source: §4.4 · p. 7

5. Defense Strategy: Subspace Camouflage

Subspace camouflage perturbs the update’s key space with semantic decoys while preserving its effect on edited keys. Evaluation shows stronger concealment can trade off against utility, with excessive camouflage degrading performance.

5.1. Subspace Camouflage via Semantic Decoy

  • The defense aims to move the update’s row space from ColSpace(K) to ColSpace(˜K), obscuring the key-space fingerprint used by the attack.The paper states that identifying the defense or recovering meaningful information about K beyond ˜K requires additional knowledge of residual matrix R.Source: §5 · p. 7, §5.1 · p. 8
  • Semantic decoys from real but unrelated subjects perturb the true key matrix, camouflaging its subspace while the update retains its effect on the edited keys.The decoy scale α controls protection strength; larger values make the reconstructed subspace more decoy-dominant and risk degrading general capabilities.Source: §5.1 · p. 8

5.2. Defense Evaluation

  • At α = 5, MEMIT’s true-subject average rank rises from 50.83 to 394.12 while efficacy and generalization remain high; stronger concealment generally sacrifices general capability.At α = 7, AlphaEdit’s efficacy drops to 0.46; Figure 4 compares protection and utility for Llama3-8B on CounterFact (N = 100).Source: §5.2 · pp. 8–9, Figure 4 · p. 9
  • For AlphaEdit, near-zero eigenvalues in its projection matrix make the defensive update more ill-conditioned, amplifying sampling-noise sensitivity and run-to-run variance relative to MEMIT.Figure 4’s shaded regions indicate standard deviation over five runs.Source: §5.2 · p. 9, Figure 4 · p. 9
  • When α ≥ 3, decoy-subject hallucination increases: TFR declines from 0.51 to below 0.43, and fluency drops from 6.23 to below 6.18.The authors describe this as a trade-off between protection and general utility.Source: §5.2 · p. 9

6. Conclusion

The conclusion presents KSTER as a reverse-engineering attack on model editing and subspace camouflage as a defense. It also reports semantic ambiguity and confused edits in existing methods.

6. Conclusion

  • KSTER analyzes parameter updates to recover sensitive data, while subspace camouflage obscures subject fingerprints in those updates.The conclusion reports high attack success and effective protection without compromising editing efficacy.Source: §6 · p. 9
  • The authors identify semantic ambiguity and confused edits in existing methods as phenomena relevant to understanding model editing’s constraints.They describe these findings as insights for future development in the field.Source: §6 · p. 9
  • The paper reports that its attack accurately recovers sensitive data and that its defense protects editing without compromising editing efficacy.Source: §6 · p. 9

Impact Statement

The impact statement frames model editing as both a leakage risk and an opportunity for auditing. It presents camouflage as a protective response and highlights further insights into editing behavior.

Impact Statement

  • Parameter updates in locate-then-edit methods can severely leak the data intended for editing, despite editing being used to erase sensitive information.Source: Impact Statement · p. 10
  • Attackers could use reverse engineering to reconstruct sensitive information from open-weight updates, while camouflage is offered to help developers safeguard edits.Source: Impact Statement · p. 10
  • The attack is also framed as a security ruler that could help authorities and third parties check whether updates match claimed corrections or conceal backdoors or malicious biases.Source: Impact Statement · p. 10
  • The authors report subject invariance in parameter space and identify semantic ambiguity and confused edits in existing methods.They connect these observations to understanding model-editing constraints.Source: Impact Statement · p. 10
  • The statement calls for research on defenses that scale without compromising utility and suggests considering edit auditability as a compliance requirement.Source: Impact Statement · p. 10

A. Limitation

The appendix notes limits in the attack’s candidate pools and in the defense’s side effects and long-term evaluation. It also describes extensions of the subject-inference attack and camouflage to ROME and AlphaEdit.

A. Limitation

  • KSTER currently relies on predefined subject pools and prompt templates; gradient-based attacks without auxiliary data are left for future work.The authors note such methods could pose a higher safety risk to model editing.Source: A. Limitation · p. 13
  • Increasing camouflage protection inevitably increases hallucination of decoy-subject knowledge, and behavior under large-scale sequential editing remains uninvestigated.Repeated camouflage could cause gradual performance degradation or latent-representation drift, potentially reducing utility.Source: A. Limitation · p. 13

B.3. Leakage Risks in Model Editing.

  • Unlike prior approaches described here, KSTER is reported to recover editing information from parameter updates without original prompts.Patil et al. (2024) relies on original prompts, while Youssef et al. (2025) does not reconstruct editing prompts and original outputs.Source: §B.3 · p. 13

C.1. Extension to ROME

  • For ROME, the update is rank one and its row space is spanned by the key vector, allowing the attack to use its subject-inference procedure with N = 1.Source: §C.1 · p. 14

C.2. Extension to AlphaEdit

  • For AlphaEdit, the null-space projection prevents recovery of some information in ColSpace(K), so the authors use a modified subject-recovery algorithm.Its scoring coefficient is defined using projection onto ColSpace(PK).Source: §C.2 · p. 14

D. Extensions of Subspace Camouflage

  • The defense is extended to ROME and AlphaEdit by retaining the decoy-key construction while changing the defensive-update constraints to match each method.For AlphaEdit, the constraint uses ColSpace(P˜K) and preserves the original update’s effect on edited keys.Source: D. Extensions of Subspace Camouflage · p. 15, §D.2 · p. 15

E.1. Detailed Experimental Setup for Attack

The attack evaluation uses candidate libraries to model limited attacker knowledge and measures recovery of edited subjects and prompts across defined recall and similarity metrics.

E.1. Detailed Experimental Setup for Attack

  • Experiments sample knowledge tuples from the first 2,000 dataset entries, using N = 1 for ROME and N ∈ {10, 50, 100} for MEMIT and AlphaEdit, averaged over five runs.The covariance and projection matrix configurations follow AlphaEdit.Source: §E.1 · p. 16
  • The subject pool draws from the first 2,000 dataset examples, and the 1,000-prompt pool combines deduplicated prompts with additional diverse prompts.The candidate libraries simulate an attacker without exact knowledge of edited subjects or prompts.Source: §E.1 · p. 16
  • The gray-box baseline ranks subjects by Jensen–Shannon divergence between pre- and post-edit logits, while its prompt procedure uses output behavior alone.The naive prompt method chooses the candidate template yielding minimum entropy on the edited model.Source: §E.1 · p. 16
  • Subject recall@N measures whether the true subject appears among the top-N candidates, while average projection coefficient measures spectral alignment with the update-derived subspace.Prompt recovery uses exact-match Top-Nr recall and SBERT cosine similarity between the true prompt and top-five recovered candidates.Source: §E.1 · p. 16

E.2. Detailed Experimental Setup for Defense

The defense evaluation tests subspace camouflage across three editors, defines editing, protection, and utility metrics, and documents example subjects and prompt templates used in the experiments.

E.2. Detailed Experimental Setup for Defense

  • The defense is evaluated on Llama3-8B-Instruct with ROME at N = 1 and MEMIT and AlphaEdit at N = 100, varying camouflage strength α and averaging five runs.Decoys are sampled for batch edits; for single edits, the decoy key vector averages keys from 10 sampled subjects.Source: §E.2 · p. 16
  • Editing quality uses greedy-decoding exact match for efficacy, rephrased-prompt generalization, neighborhood-prompt specificity, fluency, and consistency.Fluency uses n-gram entropy, and consistency uses TF-IDF cosine similarity to reference texts.Source: §E.2 · pp. 16–17
  • Protection is measured by the average rank of true edited subjects in the candidate library, while utility is assessed with F1 scores on six standard benchmarks.A higher true-subject rank indicates stronger concealment; utility benchmarks are SST, MMLU, MRPC, CoLA, RTE, and NLI.Source: §E.2 · p. 17

E.3. Subject and Prompt Template Examples for Figure 2

  • The Figure 2 examples pair named subjects such as Danielle Darrieux and Albert Einstein with relation prompts about attributes including mother tongue, birthplace, and occupation.Other listed prompt examples concern nationality, religion, place of death, alma mater, and awards.Source: §E.3 · p. 17

E.4. Generic Prompt Template Case

  • Primary experiments use the first of the listed prompt templates as the generic template, with an ablation assessing robustness to template choice.The authors state that they did not cherry-pick the templates.Source: §E.4 · p. 17

F. Additional Experiments

This section examines KSTER’s recovery of edited subjects and prompts across methods and settings, along with subspace camouflage’s effect on inference risk and utility. It also presents conditional guarantees and a proof fragment for subject recovery based on SVD and update-matrix structure.

F. Additional Experiments

  • On ROME subject inference, the white-box attack achieves 100% recall, whereas the logit-based baseline can fail on Qwen2.5-7B-Instruct.Source: F. Additional Experiments · p. 18, Table 6 · p. 19
  • ROME prompt recovery reaches Top-20 recall 1.00 for all three models with semantic similarity above 0.84, while AlphaEdit exceeds 0.95 Top-20 recall for GPT-J and Qwen2.5-7B-Instruct on CounterFact.For AlphaEdit on zsRE, semantic similarity scores of 0.77–0.89 indicate strong semantic correlation despite a slight decline versus CounterFact.Source: F. Additional Experiments · p. 18, Table 5 · p. 18, Table 7 · p. 19
  • For “Anaal Nathrakh,” the original prompt is recovered but ranks 17th behind unrelated location templates, reflecting failure to identify the musical-group category and limiting prompt recovery.Source: F. Additional Experiments · p. 18, Figure 5 · p. 19
  • For ROME, increasing camouflage α from 0 to 9 raises target-subject average rank from 1.00 to 17.20 on CounterFact and 30.40 on zsRE while efficacy stays near 1.00; at α=4, MEMIT efficacy drops to 0.59.On CounterFact, α≥3 also coincides with TFR falling from 0.51 to below 0.43 and fluency from 6.24 to less than 6.20, indicating a protection–performance trade-off.Source: F. Additional Experiments · pp. 18, 23, Table 8 · p. 20, Table 9 · p. 20
  • With batch edits (N = 100), MEMIT and AlphaEdit utility declines at excessive camouflage scales, whereas ROME with N = 1 remains notably stable.The reported protection–utility trade-off indicates that batch editing faces greater risk of degrading general capability than single-instance editing.Source: F. Additional Experiments · p. 18, Figure 6 · p. 20, Figure 7 · p. 21
  • White-box subject inference exceeds 0.94 recall across prompt templates, loses no more than 3% as candidate pools grow from 2 ∗103 to 107, and exceeds 95% recall across sequential edits.Sequential-edit tests use s=2 to 10 and report Top-(s × N) recall; template-based gray-box inference is volatile on Llama3-8B-Instruct and Qwen2.5-7B-Instruct.Source: F. Additional Experiments · p. 21, Table 11 · p. 23, Table 12 · p. 23
  • On CounterFact, prompt recovery beats the naive baseline by more than 13% in Top-1 recall and about 2% in Top-5 and Top-20 recall, while enlarging template pools from 103 to 104 reduces Top-20 recall by at most 2%.The pool-size experiments use 1,000 ground-truth templates and 9,000 generated templates.Source: F. Additional Experiments · p. 21, Table 14 · p. 24, Table 15 · p. 25

G. Theoretical Guarantees for Subject Recovery Algorithms

  • Under assumptions including candidate-set coverage, full-rank key and residual matrices, covariance invertibility, and key-vector separability, KSTER identifies the edited-subject count and exactly recovers subjects for MEMIT; the theory also characterizes the key subspace via SVD.The count-recovery result is supported by two proof approaches; the SVD result states that the column space of K equals that of VN.Source: §G.1 · pp. 24–25, §G.2 · p. 26, §2 · p. 27

2. SVD decomposition approach.

  • The proof invokes ColSpace(VN) = ColSpace(K) and Assumption G.5 to distinguish edited subjects at indices 1 through N from non-edited subjects at the remaining indices.The supplied passage ends before completing the derivation.Source: §2 · pp. 27–28

G.3. Robustness on perturbation of covariance matrix C, for MEMIT

The section analyzes subject recovery when MEMIT’s covariance matrix is perturbed and states the corresponding subspace and recovery guarantees for AlphaEdit.

G.3. Robustness on perturbation of covariance matrix C, for MEMIT

  • MEMIT analysis considers covariance matrices estimated from finite samples or another distribution, treating either perturbed estimate as C′.The section studies how such perturbation affects subject recovery.Source: §G.3 · p. 28
  • A separation margin δθ between edited and non-edited keys on one subspace becomes at least δθ − 2θmax on a second subspace.The bound relates the retained separation to the maximum principal angle between the two subspaces.Source: §G.3 · p. 28

G.4. Subject recovery for AlphaEdit, Algorithm 3

  • For AlphaEdit, the column space of the update-derived VN equals the column space of projected keys PK under Assumption G.2.Under Assumptions G.1, G.2, and G.6, Algorithm 3 is stated to output the N correct edited subjects.Source: §G.4 · p. 30

H. Theoretical Guarantees for Mechanism and Properties of Subspace Camouflage

The section establishes conditions under which subspace-camouflage updates have unique solutions, then proves properties that support the defense’s robustness: as α approaches zero, the updates recover the original editing updates, while an unknown residual makes the defense indistinguishable from original editing and leaves the true key matrix non-recoverable under stated conditions.

Defense properties

  • Subspace camouflage is presented as a general defense against subspace-based attacks, rather than a defense tailored only to KSTER.The section identifies indistinguishability and non-recoverability of the true key matrix as properties supporting this robustness.Source: H. Theoretical Guarantees for… · pp. 30–31

Theorem H.1

  • Under the stated assumptions, the three constraint problems for ROME, MEMIT, and AlphaEdit have unique solutions.The section says the solutions degenerate to the original weight updates in the zero-α limit.Source: H. Theoretical Guarantees for… · p. 31

Remark H.2

  • As α approaches zero, the camouflaged weight updates degenerate into the original updates under Theorem H.1’s assumptions.The passage states that the updated keys approach the original keys in this limit.Source: H. Theoretical Guarantees for… · p. 32

Theorem H.3

  • With unknown residual R, the camouflaged update is mathematically equivalent to an original-algorithm update using the camouflage key matrix and a new residual.The equivalence holds under the stated invertibility conditions.Source: H. Theoretical Guarantees for… · p. 33

Theorem H.4

  • Under the stated invertibility conditions, the defense update reveals virtually no further information about the true key matrix beyond the aggregated camouflage key matrix.The theorem permits almost all same-shape candidate key matrices, as the required invertibility conditions fail only on a negligible portion of possibilities.Source: H. Theoretical Guarantees for… · pp. 34–35
Loading 2602.10134v2…