Source-linked AI summary
MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents
Cheng Zhang, Xuanning Cao, Zhicheng Zheng, Hui Xiong
TL;DR
Long-term memory research has focused on recall, but dependable agents also need to forget sensitive information on request. MEMCUT evaluates that capability with annotated evidence trails and recovery metrics across memory designs and deletion baselines. Disclosure deletion reduces native recovery more than residual recovery, and subset sweeps identify candidate coverage as a bottleneck in the evaluated reference baselines.
Problem
Research has focused on accurate, efficient recall, while user-requested forgetting of sensitive information has received less attention.
Source: §1 · p. 1Method
Results
Disclosure deletion reduces native recovery far more than residual recovery; LLM-guided deletion of ten further turns from twenty retrieval candidates leaves 39–76% residual recovery.
Source: §5.2 · p. 6, §8 · p. 8Takeaways & Limitations
Subset sweeps identify candidate coverage as the main bottleneck in the evaluated reference deletion baselines.
Source: §8 · p. 8MEMCUT evaluates specified evidence trails in controlled episodes; deployment studies are needed to establish how often such trails occur and how forgetting performs on naturally occurring evidence structures.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Long-term memory has become a crucial component of large language model (LLM) agents, with most research focusing on retrieving relevant information accurately and efficiently. However, a dependable memory system must also forget sensitive user data upon request. To evaluate this capability, we introduce MemCut, a systematic benchmark for user-requested forgetting. Its 456 hand-designed secrets in long-horizon conversations have fully annotated evidence trails to test recovery after deletion, including from messages that never state the secrets directly. We propose two recovery metrics. Native recovery measures reader success through the system's retrieval, whereas residual recovery measures the same reader's success when directly given surviving objects traced to annotated disclosures or evidence. We establish three reference deletion baselines to test how far existing retrievers and write models support forgetting across five memory designs and four LLMs. Disclosure removal reduces native recovery far more than residual recovery. LLM-guided deletion of ten additional turns from twenty retrieval candidates leaves 39-76% residual recovery. Subset sweeps (144 facts, three systems, two models) show that widening the pool at a fixed budget reduces residual recovery more than deleting all original candidates. These sweeps identify candidate coverage as a bottleneck in the reference baselines.
1 Introduction
Long-term memory research has emphasized accurate, efficient recall, while user-requested forgetting of sensitive information has received less attention. MEMCUT evaluates whether protected facts remain recoverable after deletion, using annotated evidence trails and metrics that distinguish retrieval failure from surviving evidence.
1 Introduction
- MEMCUT introduces a benchmark for user-requested forgetting that tests whether protected facts remain recoverable after deletion and helps diagnose deletion outcomes.The benchmark contains 456 hand-designed secrets with annotated disclosures, evidence, look-alike records, and minimal evidence combinations.Source: §1 · pp. 1–2
- A sufficient deletion must remove the disclosure and break every supporting evidence combination, since ordinary records can reveal a secret when combined.Figure 1 illustrates that the assistant may fail to answer even while the surviving trail still supports the secret.Source: §1 · p. 1, Figure 1 · p. 1
- Native recovery tests reader success through the system’s retrieval, while residual recovery tests the same reader given surviving objects traced to annotated disclosures or evidence.The hidden-risk share, calculated from the gap relative to residual recovery, is a lower bound on residual successes missed by native retrieval.Source: §1 · pp. 1–2
- The evaluation covers five memory designs and four LLMs with three reference deletion baselines, while subset sweeps examine candidate-coverage bottlenecks.These comparisons ask how far existing retrievers and write models support user-requested forgetting.Source: §1 · p. 2
2 Related Work
Prior long-term-memory work has emphasized recall, while forgetting and unlearning benchmarks examine different settings and recovery probes. MEMCUT focuses on whether surviving records jointly reveal a secret that none states directly.
Long-term memory and recall
- Long-term memory research has largely focused on accurate and efficient recall, while similarity-based retrieval may miss evidence linked through entities rather than similar wording.Such retrieval omissions can also limit cleanup when retrieval supplies deletion candidates.Source: §2 · p. 2
Forgetting and unlearning
- Parametric unlearning and model editing target information in model weights, where consequences of an edited fact can persist.External-memory benchmarks probe updated information, reuse of obsolete facts, or direct recall after deletion.Source: §2 · p. 2
- MEMCUT tests whether surviving records jointly reveal a secret that none states directly and annotates every minimal supporting combination.The related passage contrasts this focus with benchmarks probing correlated cues and points to Table 1 for benchmark comparisons.Source: §2 · p. 2
3 The MEMCUT Benchmark
MEMCUT defines forgetting as removing a disclosure and disrupting all annotated derivations, then measures recovery through both native retrieval and surviving memory objects. Its controlled conversations combine dispersed evidence, distractors, and multiple evidence combinations to evaluate those measurements.
3.1 Task definition
- A deletion forgets a protected fact when it removes the fact’s disclosure and intersects every minimal sufficient evidence set.The task gives a deleter a natural-language request and tests recovery with a four-choice question that permits declining.Source: §3.1 · p. 2
3.2 Two recovery metrics
- Native recovery uses the system’s retrieval, whereas residual recovery gives the same reader all surviving objects derived from annotated evidence or disclosure.The hidden-risk share is the recovery gap as a share of residual recovery and is a conservative estimate of residual successes missed by native recovery.Source: §3.2 · p. 3
3.3 Construction and properties
- MEMCUT’s evidence trails support evaluation of post-deletion derivability against each secret’s annotated minimal sufficient sets.Table 2 summarizes structural properties and validation results supporting the measurements.Source: §3.3 · p. 3, Table 2 · p. 4
- The benchmark uses fifteen one-to-three-month scenarios averaging 2,893 turns, with evidence spread across sessions and about sixteen minimal sufficient sets per fact.Each fact averages eight evidence turns; about two dozen distractors help prevent topic alone from identifying evidence. The construction includes fact assembly, relation chains, aggregation, and temporal/state reasoning, and questions require correct answers under both option orders.Source: §3.3 · pp. 3–4
- Combined evidence yields 96–99% recovery in validation, while three controls yield 0%; two algorithms verify minimal cuts for all 456 facts.The dialogue is rendered from human-designed content, with authors checking every turn against the design.Source: §3.3 · pp. 3–4
4 Experimental Setup
The evaluation uses three reference deletion baselines built around existing retrievers and write models, with operations counted in source turns. Five memory designs and four write models test how retrieval unit and derivation affect forgetting.
4.1 Reference deletion baselines
- S0 deletes the disclosure and its derived objects; S1 deletes the ten highest-ranked turns from a twenty-turn retrieval pool, and S2 reranks that same pool with the write model.S1 and S2 have a ten-turn budget; neither sees the evidence annotations.Source: §4.1 · p. 4
4.2 One unit for every system
- Every memory object is traced to source turns, so pools, budgets, and deletions are counted in turns even when one object represents a whole session.Objects are deleted by closure over their source turns.Source: §4.2 · p. 4
4.3 Memory systems and models
- The five designs range from verbatim turn or session retrieval to summaries, rewritten narratives, and offline consolidation without raw text.Turn RAG and Session RAG derive nothing; MemoryBank, Nemori, and LightMem add progressively different forms of derived memory.Source: §4.3 · p. 5
- Each configuration uses one write model to write memory, score deletion candidates, and answer questions, with four models run across every system.Each fact is tested on a fresh store copy.Source: §4.3 · p. 5
5 Results: Deleting Is Not Forgetting
Deleting disclosures or additional retrieval candidates often lowers native recovery without eliminating recovery from surviving memory. The gap between these measures grows after deletion, showing that native answers can conceal recoverable secrets.
5.1 Disclosure deletion mainly reduces visibility
- Disclosure deletion (S0) lowers native recovery by 12–50 points, yet residual recovery remains 85–99%.In Turn RAG, native recovery falls from 91–95% to 57–60%; only 1.8% of evidence shares a session with the disclosure.Source: §5.1 · p. 5
5.2 Retrieval-based cleanup leaves substantial residual recovery
- 39–76% residual recovery remains after S2 deletes ten additional turns from the same twenty candidates, despite a further 7–17-point reduction from S1.S2 allows the model to choose evidence ranked below the top ten; the remaining recovery may reflect incomplete candidate coverage or imperfect selection.Source: §5.2 · pp. 5–6, Table 4 · p. 6
5.3 Deletion increases the hidden-risk share
- After deletion, Turn RAG’s hidden-risk share rises from 5–9% before deletion to 39–42% after S0 and 76–82% after S2.Across 20 configurations it ends above its starting level, exceeding one half in 14.Source: §5.3 · p. 6
- Cleanup uses the deletion request to retrieve records, while question answering uses a different query, so hard-to-retrieve evidence can survive and remain unseen.The same reader can still recover the secret when given surviving evidence-derived objects.Source: §5.3 · p. 6
6 Analysis: Why Forgetting Fails
The analysis varies candidate pool and deletion budget, then compares retrieval units, memory derivation, and fact types. Across the evaluated cleanup baselines, candidate coverage and the evidence reached by deletion are central to residual recovery.
6.1 Budget, choice, and reach
- Widening the pool from twenty to sixty at budget ten lowers S2 residual recovery by 12.5–23.9 points; deleting the whole pool of twenty still leaves 43–51% in Turn RAG and LightMem and 66–72% in MemoryBank.S2 gains diminish beyond ten deletions, and extra budget cannot remove evidence outside the pool.Source: §6.1 · pp. 6–7
- Whole-pool deletion lowers residual recovery by at most 6.5 points versus S2 at budget ten, identifying candidate coverage as the main bottleneck in these cleanup baselines.The comparison tests whether removing unselected candidates improves forgetting; recovery need not be monotonic in deletion.Source: §6.1 · p. 7
- Across 132 sweep settings, residual recovery tracks removed evidence turns closely, while system, model, and strategy together shift it by at most four points.The fitted line extrapolates to zero near k = 8.2; the pool holds only about half of a fact’s evidence on average.Source: §6.1 · p. 7
6.2 Retrieval unit and derivation
- Session RAG hides more than Turn RAG after S0: both have 96–99% residual recovery, but their hidden-risk shares are 71–82% and 39–42%, respectively.Session candidates contain fewer evidence turns and cleanup removes less residual recovery under these settings.Source: §6.2 · p. 7
- Stronger derivation does not consistently correspond to higher residual recovery: LightMem resembles Turn RAG at 39–43% versus 41–46%, while MemoryBank reaches 72–76%.MemoryBank’s lower candidate coverage accompanies its higher residual recovery; derived objects may alter which surviving evidence native retrieval exposes.Source: §6.2 · pp. 7–8
6.3 Derivation types
- Relation chains have the smallest annotated cuts at 2.7 turns, but evidence may be difficult to retrieve because it names entities without using the request’s words.A small cut therefore does not necessarily mean low residual recovery under these baselines.Source: §6.3 · p. 8
- Relation chains retain the most residual recovery after cleanup on average, with 81–94% hidden-risk shares across systems; aggregation loses 58 points to cleanup.Relation-chain evidence is linked by entities rather than request wording and has lower candidate coverage; aggregation evidence more often enters the pool.Source: §6.3 · p. 8
7 Case Study
A relation-chain fact remains derivable when cleanup deletes evidence turns without cutting every link. The case study shows how pool reach and selection jointly affect whether cleanup breaks a derivation.
7 Case Study
- The twenty-candidate pool contains only two evidence turns from different links, so deleting both leaves the four-link fact derivable.At pool size 30, S2 keeps E2 despite its potential to complete a cut with E1; at 50, deleting E5 and E6 breaks the relationship link.Source: §7 · p. 8
8 Conclusion
MEMCUT benchmarks user-requested forgetting with annotated evidence trails. Across the evaluated baselines, deletion can leave substantial residual recovery, and candidate coverage is a bottleneck.
8 Conclusion
- MEMCUT contains 456 hand-designed secrets with annotated evidence trails for evaluating forgetting in long-term agent memory.The benchmark evaluates five memory designs, four models, and three reference baselines.Source: §8 · p. 8
- 39–76% residual recovery remains after LLM-guided deletion of ten additional turns from twenty retrieval candidates.Subset sweeps identify candidate coverage as the main bottleneck in the evaluated reference baselines.Source: §8 · p. 8
- Native recovery measures success through the system’s retrieval, while residual recovery tests the same reader with surviving evidence objects.The benchmark also reports hidden-risk share to distinguish native retrieval misses from residual successes.Source: §8 · p. 8
Limitations
The evaluation uses controlled, author-specified evidence structures and reference baselines that reuse existing retrieval and write components. It studies text memory, not multimodal or parametric memory.
Limitations
- Controlled evidence trails support comparisons of candidate coverage, deletion choices, and residual recovery, but deployment studies are needed to establish how often such trails occur naturally.The episodes model recurring work, family, and social activities.Source: Limitations · p. 9
- The reference results establish performance for reusing existing retrieval and write components through a common deletion procedure, not dedicated erasure methods.MEMCUT supports evaluating methods that expand candidate coverage or change memory indexing.Source: Limitations · p. 9
- The study covers text memory; multimodal and parametric memory are outside scope, and the source-level measure omits unrelated content removed through closure or downstream utility loss.Source: Limitations · p. 9
Ethics Statement
MEMCUT uses fictional people, organizations, and events, with no real personal data. The authors note possible misuse of reconstruction methods and describe an independent annotation audit.
Ethics Statement
- All people, organizations, and events in MEMCUT are fictional; the benchmark contains no real personal data.Source: Ethics Statement · p. 9
- The authors state that describing reconstruction of deleted secrets could be misused, while arguing that measuring the risk is a prerequisite for reducing it.Source: Ethics Statement · p. 9
- Two graduate NLP students unaffiliated with benchmark construction audited the annotations, each working about 25 hours after being told the content was fictional.The benchmark will be released for research use.Source: Ethics Statement · p. 9
Use of AI Assistants
Across the supplied results, widening candidate pools improves forgetting more than increasing deletion budgets in the stated comparison, while residual recovery remains tied to evidence reached by deletion. The appendix also reports recovery patterns, dataset structure, retrieval-unit effects, and evaluation boundaries.
A Dataset Details
- The 456 protected facts appear across fifteen long episodes totaling 43,402 turns in 490 sessions; their 7,148 minimal sufficient sets each contain two to five turns (mean 2.7).A fact’s evidence has a median first-to-last span of 2,259 turns and 29 days.Source: A Dataset Details · p. 11
B.1 Recovery at every deletion level
- Native recovery is 82–96% before deletion except Session RAG at 48–70%; hidden-risk share rises from Full to S2 in all 20 configurations, while S0 closure can remove summarized evidence.Source: §B.1 · pp. 11–12, Table 8 · p. 13
B.2 Retrieval unit and derivation strength
- Session RAG contexts average 13.3k tokens and 1.9 evidence turns versus Turn RAG’s 1.1k tokens and 4.8; after cleanup, Session RAG leaves 10–23 points more recoverable.Its candidate pools are diluted by non-evidence turns from the same sessions.Source: §B.2 · p. 13
B.3 Where cleanup loses the evidence
- At the main pool of twenty, candidates contain 4.5 evidence turns in Turn RAG and LightMem and 3.0 in MemoryBank; widening to sixty still leaves evidence unreached.S2 deletes 4.4, 4.3, and 2.7 evidence turns, respectively, at pool twenty.Source: §B.3 · p. 13
- Relation chains lose 2.4 evidence turns per fact under S2, fewer than fact assembly (3.8), temporal state (3.9), and aggregation (4.6).Their evidence shares little wording with the request, so similarity retrieval ranks it low.Source: §B.3 · p. 14
- S2’s gain over S1 grows from 5–15 points at pool twenty to 20–32 at pool sixty, reflecting judgment over candidates beyond the top ten.Source: §B.3 · p. 14
B.4 Deletion cost and residual recovery
- 12.50 points: widening the pool from twenty to sixty at budget ten lowers residual recovery and removes 1.04 fewer non-evidence turns than the comparison setting.Source: §B.4 · p. 15
- These results do not establish downstream utility preservation; evidence turns may support legitimate tasks, and closure can remove unrelated content from shared objects.Retained task performance remains a separate evaluation dimension.Source: §B.4 · p. 15
C Leakage by Derivation Type
Across derivation types, relation chains remain hardest to recover after cleanup, while aggregation shows the lowest residual recovery. The case study illustrates how retrieval-pool coverage can limit deletion, and the related-work discussion places external-memory forgetting alongside work on unlearning and contextual privacy.
C Leakage by Derivation Type
- Relation chains have the highest residual recovery in 12 of 16 configurations, while aggregation has the lowest in all 16.Table 13 gives recovery by derivation type across three memory systems with DeepSeek and Qwen.Source: C Leakage by Derivation Type · p. 15, Table 13 · p. 16
- Relation chains are least answered before deletion (81% versus 90–96% for other types), and deleting the disclosure costs them 63 points of native recovery.In LightMem at S0, native recovery is 30% for relation chains, versus 94% for fact assembly and 90% for temporal questions; their evidence uses people and firm names rather than request wording.Source: C Leakage by Derivation Type · pp. 15–16
- Replacing retrieval ranking with LLM judgment lowers residual recovery by 14 points for fact assembly and aggregation, 11 for temporal facts, and 8 for relation chains.The smaller relation-chain gain is consistent with limited candidate coverage, since judgment selects only turns already in the pool.Source: C Leakage by Derivation Type · p. 16
- Session RAG answers fewer questions than Turn RAG after S0 for every type, with the largest gap for aggregation (about 54 points) and the smallest for relation chains (17 points).The text links aggregation's larger loss to sessions rarely containing all required values; relation chains lose least because Turn RAG already fails to surface them.Source: C Leakage by Derivation Type · p. 16
- After S2, hidden-risk shares for fact assembly, aggregation, and temporal facts fall from 68–85% in Turn RAG and MemoryBank to 20–39% in Nemori and LightMem, while relation chains remain at 81–94%.The authors suggest within-topic evidence combination in derived objects and cross-topic links for relation chains as possible explanations.Source: C Leakage by Derivation Type · p. 16
- Aggregation residual recovery falls 15 points on average when only the disclosure is deleted, versus at most 3 for other types, despite its evidence remaining sufficient by design.The text attributes this to readers computing aggregates less reliably and, in derived systems, summaries that combine the disclosure with other values being removed under closure; it also notes possible collateral deletion and cases where native recovery slightly exceeds residual recovery.Source: C Leakage by Derivation Type · pp. 16–17
D Case Study Details
- In the case study, similarity ranks the seven evidence turns from 3rd to 141st, and a 20-turn pool contains only E1 and E3, leaving the fact derivable after S2.S2 first makes the fact no longer derivable at a pool size of 50, when E6 joins the deleted turns and completes a cut.Source: D Case Study Details · p. 17
- Entity expansion could place a complete cut within a ten-turn pool in the example, but its effectiveness as a deletion strategy remains untested.The evidence turns are each one hop from an entity in the request.Source: D Case Study Details · p. 17