AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents

1 Introduction

• The benchmark compares exposed-then-deleted agents with matched never-exposed controls, attributing exact post-deletion recovery to the original exposure rather than guessing or background knowledge.A correct reconstruction in the exposed world counts as a disclosure regardless of the control rate.Source: §1 · p. 1
• A single direct question recovers the deleted target from Qwen2.5-7B-Instruct in roughly two thirds of cases, while matched never-exposed controls have zero recovery.In a separate experiment, exact reconstruction also occurs when only separately stored components remain under a public composition rule, across Qwen, Mistral, and Phi-3.Source: §1 · pp. 1–2
• A target-bearing raw reply can be scored as failure when the requested format and official parser are misaligned, so reconstruction, write-back, and fresh-session recall are measured separately.Disclosure can occur without a new write, and blocking a write cannot erase information already in memory.Source: §1 · p. 2
• Removing or rebuilding tracked summaries linked by complete provenance yields zero observed recovery in tested positive-baseline settings, while an exact-literal write filter blocks reinsertion but not existing residue or earlier disclosure.The provenance result does not establish general semantic erasure.Source: §1 · p. 2
• The benchmark offers a controlled audit for synthetic exact-string targets, controlled memory constructions, finite query strategies, and specified response interfaces, not a general solution to agent forgetting.Semantic deletion, incomplete provenance, learned or compressed memory, natural-language secrets, and autonomous production memory systems remain open.Source: §1 · p. 2

2 Related Work

• Unlike machine-unlearning and parameter-editing work, AfterDeleteBench holds weights fixed, deletes external agent state, attributes recovery with a paired control, and separates reconstruction from later write-back and fresh-session recall.The related work includes extraction studies and controlled LLM-unlearning evaluations.Source: §2 · p. 2
• Prior agent-memory and auditing studies motivate behavioral privacy tests, but the cited works do not use the exposed-then-deleted versus never-exposed reconstruction contrast.They include long-term recall evaluation, memory-management studies, private-record extraction, and persistent-memory attack research.Source: §2 · p. 2
• Concurrent work also examines external-state deletion, provenance, and behavioral leakage, so the paper limits its novelty claim to a specific combination of audit and intervention measurements.That combination includes paired attribution, verbatim and non-verbatim reconstruction, interface diagnostics, lifecycle separation, and a matched write-effect intervention.Source: §2 · p. 3
• The provenance intervention retires and rebuilds dependent summaries using recorded source identifiers, while the exact-literal filter checks a proposed new literal before storage; neither is presented as a general solution to forgetting.The paper explicitly does not claim that provenance metadata captures every semantic influence.Source: §2 · p. 3

3 Problem Setup and AfterDeleteBench

• Each profile contains a structured target, a public attribute naming it, background facts, and matched target-bearing and neutral exposure records; derived persistent state is a persistent representation later produced from source records.A source record is an individually addressable interaction committed to external memory.Source: §3 · p. 3
• The forget world exposes the model to the target and deletes that source after consolidation, while a matched clean world substitutes a neutral interaction and follows the same workflow without revealing the target.The worlds share background records, model, configuration, memory architecture, auditor, query budget, and scoring rule; the model does not see the original exposure or target truth.Source: §3 · p. 3
• The primary reconstruction endpoint is exact equality between the auditor’s submitted prediction and the target, and the paired recovery difference compares forget-world and clean-world rates.The difference attributes recovery to exposure within this paired construction, but does not estimate normal retrievability because there is no exposed-and-retained reference condition.Source: §3 · pp. 3–4
• Profiles span four synthetic domains and four target structures, with 200 profiles per split and a generator-held-out shift that retains the same domains, structures, and core composition.The shift holds out target families, templates, clue frames, and entity vocabulary; it is not unrestricted real-world shift.Source: §3 · p. 4
• Hierarchical memory retains raw records and an immutable extractive summary, so exact source deletion removes the addressed entry but deliberately leaves the distinct summary untouched.The model receives the ordered post-deletion memory, making the primary construction an inspectable test of recovery from a known derived descendant rather than every production architecture.Source: §3 · p. 4
• Recorded-lineage closure retires dependent summaries and rebuilds them from surviving records, while a separate exposure-group deletion diagnostic removes declared neighboring records and summaries containing removed text verbatim.Complete recorded provenance makes closure an idealized structural upper bound, not evidence of semantic forgetting in untracked or learned memory.Source: §3 · pp. 4–5
• The transformed-memory suite tests full literals, separately stored components under a public composition rule, closure, and a lossy condition omitting the number.The exact-literal gate instead checks a proposed write against a protected descriptor without normalization, and cannot remove existing state or undo disclosure.Source: §3 · p. 5
• The official scorer parses the last visible reply as JSON and counts only a literal target match in a top-level string value as reconstruction; parser mismatches can conceal target-bearing replies.The audit uses a shared set of seven query templates with at most ten queries; commitment tracks durable writes and activation tests exact production in a fresh session after transcript and auditor state removal, while E is descriptive rather than causal.Source: §3 · pp. 4, 6

4 Experimental Setup

• Qwen2.5-7B-Instruct is the primary model, while Mistral-7B-Instruct-v0.3 and Phi-3-medium-4k-instruct appear in cross-model interface and transformed-memory studies.The primary Qwen matrix does not run every combination, so auditor comparisons are restricted to their shared hierarchical-memory, exact-deletion, gate-off block.Source: §4 · p. 6
• Bounded Qwen studies vary memory serialization, retrieval exposure, and activation wording, while cross-model studies test how response-interface effects shape apparent recovery failure.The frozen Qwen-facing interface, including its mismatch, was transferred to Mistral and Phi-3 before development-based model-specific interface selection and held-out evaluation.Source: §4 · p. 6
• A second alignment suite uses 104 IID and 104 generator-held-out OOD profiles per model across the four original structures, with modules for contract alignment, matched write effects, and transformed-memory reconstruction.A Qwen-only diagnostic changes only the activation contract on frozen matched states; studies beyond the primary matrix are post-hoc relative to it, though each design or selected configuration was frozen before held-out inference.Source: §4 · p. 7
• Nonboundary endpoints use 95% percentile intervals from 10,000 paired-profile bootstrap resamples, with no pooled estimator across models or splits and no p-values.The 100-case utility check is a sanity check for obvious benign-behavior corruption, not comprehensive downstream validation.Source: §4 · p. 7

5.1 Exact source deletion leaves substantial recoverable residue

• A single DIRECT query recovered targets with ΔRec = 0.680 on IID profiles and 0.640 on OOD profiles; clean recovery was 0/200 in both splits.The measured channel was located in retained hierarchical state in this controlled construction.Source: §5.1 · p. 7, Table 1 · p. 8

5.2 Recovery persists without a surviving complete literal

• With no contiguous target in 1,248 states, C1 exact-recovery gaps ranged from 0.615 to 0.990; C2 closure zeroed recovery and component disclosure, while C3 still exposed retained components.The C3 omitted number occurred in 0/104 profiles in every tested cell, while complete-target recovery was zero.Source: §5.2 · pp. 7–8, Table 2 · p. 8

5.3 Deletion and write controls act at different lifecycle stages

• Closure yielded zero observed R, C, A, and E gaps; the gate blocked C but left R unchanged and A positive, while write-allowed raised Qwen activation.For Qwen profiles with R = 1, write-allowed activation rose from 64/76 to 76/76 IID and from 58/75 to 69/75 OOD; effects varied across models.Source: §5.3 · p. 8

5.4 Response contracts can hide disclosure and reverse measured effects

• Aligned contracts produced substantial measured recovery for Mistral and Phi-3, while legacy contracts scored zero despite target-bearing raw replies; Qwen’s measured write effect could reverse.Changing only Phi-3’s activation contract sent official E from 77/104 to 0/104 IID, while raw replies still contained the target in 78/104 cases.Source: §5.4 · p. 9

5.5 Most observed recovery appears on the first productive probe

• 3,081 of 3,088 observed recovery events occurred on query 1; scripted, greedy, and frozen-PPO auditors tied at ΔRec = 0.735 IID and 0.745 OOD.Specified serialization, retrieval, and activation-wording variations retained positive findings, but the saturated 100/100 utility scores and 20/20 near-miss passes do not characterize realistic downstream costs or false blocking at scale.Source: §5.5 · p. 9

• Exact source-record removal can leave information available through derived state, while a write-time safeguard cannot undo an earlier disclosure.The paper distinguishes disclosure from later write-back and fresh-session recall.Source: §6 · p. 10
• Non-verbatim reconstruction extends the tested failure mode beyond surviving complete copies, but controlled targets and deterministic transforms do not establish the same behavior for natural-language secrets or learned representations.When one component was omitted, complete reconstruction fell to zero in tested cells while remaining components were still exposed.Source: §6 · p. 10
• The scope is limited to synthetic exact-string targets, a controlled external-memory harness, deterministic transformations, a finite operator vocabulary, and generator-held-out synthetic shift.Natural-language and semantic secrets, learned or compressed memory, incomplete provenance, independent production systems, and realistic runtime or storage costs remain open.Source: §6 · p. 10

7 Conclusion

• AfterDeleteBench shows that source deletion need not make previously exposed information behaviorally unavailable, including when the target must be composed from separately stored components.The conclusion separates reconstruction, write-back, and fresh-session recall, and describes provenance-aware closure under complete recorded lineage alongside an exact-literal filter against future reinsertion.Source: §7 · p. 10

Ethics statement

• The study uses synthetic secrets and controlled memory, and its results do not justify classifying an untested agent as safe or unsafe.Source: Ethics statement · p. 10

Reproducibility statement

• The reproducibility statement says frozen reports, validation records, tables, and provenance artifacts accompany repository evidence, with code to be made public after acceptance.Source: Reproducibility statement · p. 11

AI Use Statement

• The authors used generative AI for code development and manuscript editing, while retaining responsibility for the reviewed and verified work.Source: AI Use Statement · p. 11

A Extended Threat Model and Lifecycle

• The lifecycle distinguishes clean and forget worlds, persistent states before and after deletion, and a branch-specific candidate-write transition.The fixed fresh-session probe produces a later activation outcome.Source: A Extended Threat Model and Lifecycle · p. 14
• The end-to-end endpoint is a conjunction, not a causal write-effect estimator; the matched write diagnostic compares activation under write-allowed versus no-write conditions.A surviving residue can produce activation even when commitment is zero.Source: A Extended Threat Model and Lifecycle · p. 14

B Auditors and Correction History

• The expert script uses a fixed prior ordering, while greedy information gain updates a Beta-style productivity belief over public replies.The contextual bandit uses a frozen disjoint LinUCB policy over seven probes and the same 24 public features used by PPO.Source: B Auditors and Correction History · p. 14
• The prompted auditor must return a single-field operator JSON response; invalid or unavailable choices trigger a frozen-prior fallback without retry.Source: B Auditors and Correction History · p. 15
• After a prompted-auditor stub was found defective, only four affected cells were rerun; the defective arm contributes to no reported result.Validation covered 800 replacement rows and 400 pairs.Source: B Auditors and Correction History · p. 15

C Experiment Chronology

• The chronology distinguishes evidence roles; later confirmations were frozen before their own inference but remain post-hoc relative to the original study.Source: C Experiment Chronology · p. 15, Table 4 · p. 16

D.1 All auditor configurations

• Table 5 reports primary Qwen ΔRec with 95% paired intervals in the matched hierarchical, exact-deletion, gate-off block; clean recovery is zero throughout.Source: Table 5 · p. 16

D.2 Structure and domain breakdowns

• 0/50 clean and forget exact recoveries were recorded by all eight auditors for compositional profiles in both splits.The analyses are post-hoc and descriptive.Source: §D.2 · p. 16

D.3 Lifecycle interventions

• Recorded-lineage closure acts on existing derived state, whereas the exact-literal gate acts only on proposed writes; activation may remain positive when commitment is zero.The figure's boundary markers are observed endpoints, not certainty about population values.Source: §D.3 · p. 17, Figure 4 · p. 18

E Cross-Model Interface Evidence

• Historical one-turn DIRECT official recovery was zero for Mistral and Phi-3 in both splits, but raw replies were not retained; later calibrated held-out gaps were positive.The later confirmation panel found target-bearing raw replies despite official zeros under legacy contracts.Source: E Cross-Model Interface Evidence · p. 18

F Specified Qwen Variations

• The specified Qwen variations cover representation, retrieval, and activation wording; the fact ledger is canonical JSON, and these variations are not representative of arbitrary systems.Source: F Specified Qwen Variations · p. 19, Table 7 · p. 19, Figure 6 · p. 20

G Second Alignment Suite

• The second alignment suite was frozen before inference on its new panels but remains post-hoc relative to the original study; its package validation reported zero findings.The package contains 2,496 validated rows per model package, not 2,496 model calls.Source: G Second Alignment Suite · p. 21

G.1 Full-lifecycle contract factorial

• 77/104 Mistral and 78/104 Phi-3 legacy IID raw replies contained the target despite official zeros, showing the contract alignment issue in the confirmation panel.Source: §G.1 · p. 21

G.2 Matched write/no-write diagnostic

• 0.115 IID and 0.096 OOD are the full-denominator Qwen write-allowed-minus-no-write activation differences.The OOD conditional contrast is 11/75, while the full-denominator contrast is 10/104 because one R = 0 profile activates under no-write.Source: §G.2 · pp. 21–22, Table 9 · p. 22
• Under the legacy contract, Qwen IID write-allowed official activation falls despite target occurrence in every raw reply, while OOD activation rises modestly.On matched reconstructed states, no-write and gate-blocked replies are identical; these are new episodes, not the historical Table 6 runs.Source: §G.2 · p. 22

G.3 Transformed-memory diagnostics

• Complete-target reconstruction is zero in the tested transformed-memory cells, and C2 has zero observed component occurrence in every cell.The C3 table reports literal component occurrence in forget-world replies; it is distinct from the exact-target endpoint.Source: §G.3 · p. 22, Table 11 · p. 23

H Utility and Operational Scope

• The 100-case behavioral utility check scores 100/100 under baseline deletion, closure, and gate-on, but does not establish realistic downstream utility or collateral over-pruning.All scored targets have support independent of the deleted source; among 20 gate near-misses, 0/20 are falsely blocked, with an exact marginal upper bound of 0.168.Source: H Utility and Operational Scope · p. 23

I Reproducibility Details

• Each model call uses a private prompt with six keys, and the official scorer accepts a nonempty top-level string value.Audit and activation calls use the selected response contract; activation starts with an empty audit transcript and carries forward only post-write persistent memory.Source: I Reproducibility Details · pp. 23–24
• Implementation identifiers may retain “victim” for the evaluated language model, while the manuscript calls it the model or evaluated model.The auditor is the black-box querying policy; the evaluator is benchmark-side logic holding target truth and computing scores.Source: I Reproducibility Details · p. 24
• Primary memory rendering joins active raw records in lexicographic entry_id order, while exact deletion removes the addressed raw entry but leaves its preexisting summary.Recorded-lineage closure removes summaries naming that entry and rebuilds from surviving active records; fresh sessions clear audit history and policy state.Source: I Reproducibility Details · p. 24
• Inference is pinned to Transformers 4.43.1 and PyTorch 2.4.0, with bfloat16, greedy decoding, at most 96 new tokens, and seed 20260809.The passage also records generator and bootstrap seeds and frozen artifact revisions, package bindings, and checksums.Source: I Reproducibility Details · p. 24
• The primary matrix repeats evaluations over the same 200 profiles per split, so evaluation-unit totals are not independent profile counts.Nonboundary intervals resample paired profiles 10,000 times; marginal exact binomial intervals are reported separately.Source: I Reproducibility Details · p. 24

J Extended Limitations and Future Work

• The externally controlled memory construction supports internal attribution but has limited external architecture validity because the model does not choose memory operations.Independent memory frameworks, learned abstractions, semantic memory, and independent retrieval or consolidation pipelines remain to be studied.Source: J Extended Limitations and Future Work · p. 24
• Exact structured targets do not establish behavior for paraphrasable facts, preferences, or concepts; semantic evaluation needs a separately specified scorer.The paired clean control attributes recovery to exposure but does not measure the fraction of normally retrievable information preserved.Source: J Extended Limitations and Future Work · pp. 24–25
• Complete recorded provenance makes closure an idealized structural upper bound in the deterministic extractive harness, while missing or untracked dependencies can evade it.The exact-literal gate recognizes only an exact protected representation; realistic downstream utility and collateral over-pruning remain uncharacterized.Source: J Extended Limitations and Future Work · p. 25