Source-linked AI summary
Suppressed, Not Erased: A Representational Trace of Edited Facts Survives Even Weight-Free Knowledge Editing
Priyansh Srivastava, Romit Chatterjee
TL;DR
Knowledge-editing benchmarks test whether models produce new facts but not whether the original facts remain decodable internally. The paper probes edited hidden states across three editors and finds the original object remains linearly decodable after successful edits, including under a weight-free editor. It interprets editing in this setting as suppression rather than erasure.
Problem
Knowledge-editing evaluations test production of the new fact near an edit but leave open whether the original fact is truly gone.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
For deletion audits, report how recoverable suppressed content remains alongside efficacy; the GRACE result locates a residual trace beyond weight changes.
Source: §7 · p. 5The study covers GPT-2-XL and single edits; linear probing detects linearly decodable traces, and GRACE’s estimate uses only 14 of 50 facts.
Source: §6 · p. 5, Limitations · p. 6
Abstract
from arXiv · showhide
Knowledge-editing benchmarks certify local correctness, whether an edited model produces the new fact on near-edit prompts but not how much of the original fact remains decodable inside the model. We study residual knowledge directly with a linear trace probe: after editing a fact, we ask whether the original object is still recoverable from the model's hidden states. On GPT-2-XL, across three mechanistically distinct editors applied to 50 CounterFact edits, the original object remains linearly decodable well above chance after a successful edit (probe accuracy 0.96 for ROME, 0.86 for constrained fine-tuning, and 0.79 for the memory-based editor GRACE, against a chance level of 0.50; all edits reach 100% generation-based success). The GRACE result is the most informative: GRACE changes zero base-model weights, overriding the fact through an external memory, yet the original object is still decodable from the underlying network, so the residual trace cannot be attributed to an incomplete weight update. We read this as evidence that editing, even when behaviorally successful, suppresses rather than erases the original association in representational space. We also report a relearning-savings instrument that did not behave reliably in our setting and discuss why; we treat it as a negative methodological result rather than evidence. Code and data are released.
1 Introduction
Knowledge-editing evaluations test whether the new fact appears, but leave open whether the original fact remains internally decodable. This study probes that residual trace across editors, including a weight-free memory editor, and reports its scope and a relearning-savings instrument that proved unreliable.
1 Introduction
- Standard editing metrics certify that the new fact appears near the edit, not whether the original fact is truly gone.Prior extraction work recovered answers from ROME-edited GPT-J about 38% of the time using hidden-state and rephrasing attacks.Source: §1 · p. 1
- A linear trace probe tests whether the original object can be recovered from edited-model hidden states.The paper presents its contribution as empirical and deliberately narrow.Source: §1 · p. 1
- After 100%-successful single edits, the original object remains linearly decodable well above chance across three mechanistically distinct editors on GPT-2-XL.The memory-based editor GRACE changes no base-model weights, yet the original object remains decodable; the authors interpret this as suppression rather than erasure.Source: §1 · p. 1, Figure 1 · p. 2
- GRACE’s residual trace is not attributable to an incomplete weight update because GRACE changes no base-model weights.The authors use this control to locate the trace in the underlying representation left intact by the edit.Source: §1 · p. 1
- The relearning-savings instrument did not behave reliably in this setting, and the authors analyze why memory-based editing breaks its assumptions.They present it as a negative methodological result rather than evidence.Source: §1 · pp. 1–2
- The audit covers one canonical testbed, GPT-2-XL, and single-edit residual traces.The paper explicitly limits the scope of the result.Source: §1 · p. 2
2 Related Work
Prior work studies editing efficacy, extraction of residual answers, and traces of unlearned knowledge. This paper follows the extraction line while comparing distinct editors and using a weight-free editor to examine where residual traces persist.
2 Related Work
- Standard editing evaluation measures efficacy, paraphrase, and specificity, while audits report fragility under generation-based evaluation and failures to propagate edits to entailed facts.This paper turns from whether the new fact is present to whether the old fact remains internally decodable.Source: §2 · p. 2
- Prior extraction work recovered ostensibly deleted answers from ROME-edited models, while other work traced unlearned knowledge in parameter space or restored suppressed capabilities through relearning attacks.The cited relearning-attack studies use small corpora.Source: §2 · p. 2
- Unlike prior extraction work, this study fixes behavioral success at 100%, compares editors with different mechanisms, and includes a memory-based editor that changes no weights.The weight-free editor lets the authors ask where the residual trace lives, rather than only whether it exists.Source: §2 · p. 2
- Mechanistic analyses suggest edits may add suppressive structure instead of removing the original association; this representational result is consistent with that view.The paper offers an editor-agnostic behavioral-plus-probing measurement.Source: §2 · p. 2
3 Method
The method replaces a true subject–relation–object fact with a counterfactual target using three editors, then tests whether the original or counterfactual object dominates in edited hidden states. It also measures behavioral resurfacing and relearning savings, though the latter is treated as unreliable in this setting.
3 Method
- The edit replaces a true subject–relation–object triple with a counterfactual target object.The setup illustrates the replacement with the Eiffel Tower’s location changing from Paris to Rome.Source: §3 · p. 2
- The experiment applies single CounterFact edits to GPT-2-XL with ROME, constrained fine-tuning, and GRACE, using generation-based success judgments across the distinct editor mechanisms.ROME and FT-L alter weights, whereas GRACE uses an external override without changing base weights.Source: §3 · pp. 2–3
- The linear trace probe asks whether the original object still linearly dominates the counterfactual object in the model’s representation.Above-chance accuracy detects linearly separable, layer-local signal; it cannot rule out a trace when accuracy is low, and non-finite hidden states are excluded.Source: §3 · p. 3
- For each edited fact and candidate layer, the probe compares hidden-state projections onto the original and counterfactual object directions, then predicts which object’s projection is larger.The authors train cross-validated logistic regression on standardized hidden-state features and report best-layer accuracy against chance of 0.5.Source: §3 · p. 3
- The resurfacing rate measures how often the original object reappears in generation on a small stress set of paraphrase and long-context prompts.The authors describe it as a strict generation-level signal and report it for completeness.Source: §3 · p. 3
- The relearning-savings score compares the optimization cost of restoring the original fact with teaching a matched novel fact, but proved unreliable in this setting.The authors do not draw conclusions from the instrument.Source: §3 · p. 3
4 Results
Across three editors, the original object remains linearly decodable after every edit succeeds behaviorally. The above-chance trace also appears with GRACE, which leaves base-model weights unchanged.
4 Results
- GRACE’s 0.79 probe accuracy despite zero base-weight change supports a residual trace in the underlying representation, not an incomplete weight update.ROME and FT-L change targeted weights by approximately 1.1 × 10^4 and 3.2 × 10^4, respectively.Source: §4 · p. 4
- 0.01–0.10 is the reported resurfacing-rate range; the original object rarely reappeared verbatim in generation despite remaining linearly decodable.The authors distinguish representational persistence detected by the probe from behavioral re-expression and do not rely on resurfacing rate as evidence.Source: §4 · p. 4
- 0.96 for ROME, 0.86 for FT-L, and 0.79 for GRACE are probe accuracies for recovering the original object, against a 0.50 chance level, after successful edits.All 50 edits reached generation-based success for every editor.Source: §4 · pp. 3–4, Table 1 · p. 4
5 The Savings Instrument: A Negative Result
The relearning-savings instrument did not behave reliably in this setting, so the authors do not use it as evidence. Coarse cost measurement and GRACE’s active override undermine its interpretability.
5 The Savings Instrument: A Negative Result
- GRACE’s mean savings of approximately −5.2 reflects restoration hitting the step cap against an active override, not the absence of a latent trace.For ROME and FT-L, savings were approximately 0.00 and −0.30 and were also uninterpretable at the available granularity.Source: §5 · p. 4
- Checking restoration every five optimization steps made the smallest distinguishable cost five steps, too coarse for the targeted effect sizes.Source: §5 · p. 4
- The authors exclude savings as evidence and identify two needed changes: finer cost resolution and deactivation of external overrides during restoration.They leave a corrected savings instrument to future work and rely on the probe instead.Source: §5 · p. 4
6 Caveats by Editor
The editor-specific caveats qualify the precision and interpretation of the probe and behavioral instruments. GRACE has the clearest sample-size constraint, while resurfacing is unreliable across editors.
GRACE
- 36 of 50 facts were excluded for GRACE, leaving its probe based on 14 facts and making the 0.79 estimate imprecise.The above-chance trace remains a qualitative finding, but the value should not be read as a clean N = 50 estimate.Source: §6 · p. 5
ROME
- ROME's probe uses all 50 facts, but the reported results do not establish whether its trace is localized to the edited layer or distributed.The paper does not report a full per-layer profile.Source: §6 · p. 5
FT-L
- FT-L's savings result is uninterpretable in part because editing and restoration fine-tune the same weights, alongside a granularity issue.The reported savings value is approximately −0.30.Source: §6 · p. 5
Across all three
- Resurfacing is low and noisy across editors (0.01–0.10), so the paper treats it as evidence neither for nor against a trace.It is reported for completeness only.Source: §6 · p. 5
7 Discussion
The discussion argues that edit audits should measure what remains recoverable, not only whether the new fact appears. The GRACE control supports interpreting the residual trace as representational persistence rather than merely incomplete weight editing.
7 Discussion
- A behaviorally successful edit is an incomplete certificate when the original object remains linearly decodable.For privacy- and safety-motivated deletion, the paper identifies recoverability of suppressed content as the quantity that matters.Source: §7 · p. 5
- Because GRACE changes no weights yet leaves an above-chance trace, the result is not merely an incompleteness of weight editing.The paper interprets the trace as a property of the representation that editors operate on top of.Source: §7 · p. 5
- The discussion suggests that genuine deletion methods may need to target representations directly rather than only overwrite or override output mappings.Source: §7 · p. 5
8 Conclusion
The conclusion reports above-chance recovery of original objects after behaviorally successful edits, including under a weight-free editor. It also treats the unreliable savings instrument as a methodological failure and identifies broader audits and repaired instruments as future work.
8 Conclusion
- Across GPT-2-XL, three editors, and 50 CounterFact edits, the original object remains linearly decodable after successful edits, including under weight-free GRACE.The conclusion characterizes the result as suppression rather than erasure in representational space.Source: §8 · p. 5
- The relearning-savings instrument could not be made reliable and is reported as a failure, not as evidence.Source: §8 · p. 6
- The conclusion identifies extending audits to more models and sequential editing, and repairing the savings and resurfacing instruments, as next steps.Source: §8 · p. 6
Limitations
The study limits its claims to single-edit traces on GPT-2-XL and 50 English encyclopedic CounterFact edits across three editors. Its linear probe detects linearly decodable signal, and GRACE's estimate is constrained by excluded facts.
Limitations
- All results concern GPT-2-XL and single edits; the study makes no claim about larger or instruction-tuned models, sequential editing, collateral damage, or identity drift.The data scope is three editors and 50 English encyclopedic CounterFact edits.Source: Limitations · p. 6
- Above-chance probe accuracy establishes a linearly decodable trace, but a lower value would not prove absence of information.It would show only linear undetectability.Source: Limitations · p. 6
- GRACE's probe rests on 14 of 50 facts because its hook produced non-finite hidden states, leaving its precise value uncertain.The savings instrument failed in this setting, and the sparse resurfacing signal is not relied on.Source: Limitations · p. 6
Ethics Statement
The paper frames its trace probe as a dual-use measure of residual knowledge and limits this study to public models and public encyclopedic facts.
Ethics Statement
- The trace probe has dual-use potential: it can audit deletion completeness but could also guide recovery.The authors describe honest deletion claims as requiring an audit of completeness.Source: Ethics Statement · p. 6
- The study edits and recovers no personal or hazardous information, using only public models and public encyclopedic facts.Source: Ethics Statement · p. 6
- The authors release code oriented toward auditing whether deletion is complete.Source: Ethics Statement · p. 6
A Implementation Details
The implementation applies three editing approaches at a fixed mid-layer down-projection, then probes hidden states and evaluates generation-based edit success.
A Implementation Details
- ROME uses a rank-one update from a real intermediate activation, FT-L uses constrained gradient steps, and GRACE overrides through a forward hook without changing base weights.FT-L uses gradient clipping and upcasts trained weights to full precision during updates for numerical stability under half-precision inference.Source: A Implementation Details · p. 7
- The logistic-regression probe uses standardized hidden-state features, cross-validation, and the best-performing candidate layer at the edit prompt.The probe is trained per candidate layer.Source: A Implementation Details · p. 7
- Edit success is measured by greedy-decoding generation with alias matching.Source: A Implementation Details · p. 7
- An append-only ledger makes the pipeline resumable for checkpointing and reproducibility, and the code and result ledger are released.Source: A Implementation Details · p. 7