PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

Bofan Xie, Chaokun Wang, Guanchen Ge, Songyao Wang, Fang Niu, Hang Zhang, Wei Wang

arXiv:2610.18052v1cs.AI

TL;DR

Existing agent-memory benchmarks largely assess storing and retrieving information, leaving explicit user-requested forgetting under-evaluated. BEMULA evaluates memory unlearning with paired dialogue versions and five probe suites. Across evaluated systems, forget requests almost never pass end to end, leaving memory unlearning an open challenge.

  • Problem

    Existing agent-memory benchmarks primarily test retention and use, leaving whether agents can forget information on explicit user request under-evaluated.

    Source: §1 · pp. 1–2

  • Method

    BEMULA pairs each dialogue with a control version and evaluates 50 dialogue instances using 1,742 questions across five suites.

    Source: §1 · p. 2, §3.2.5 · p. 6

  • Results

    Forget-request pass rates never exceed 1.5% (LangMem on DeepSeek-V4.1-Flash), and most systems remain at zero.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    Memory unlearning remains an open challenge because systems retain target information reliably, yet forgetting on demand largely fails.

    Source: §5 · p. 9

    The dialogues are synthetic and come from one generation pipeline, and the benchmark covers conversational memory rather than tool-using agents.

    Source: §5 · p. 9, ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users increasingly ask them to delete sensitive or outdated facts. A memory system should honor such requests. It should remove exactly what was requested and keep the rest of its memory intact. However, existing memory benchmarks evaluate only memory writing, retrieval, and organization, so whether agents can forget at the user's explicit request remains under-evaluated. To address this gap, we propose Bemula, the first benchmark dedicated to memory unlearning. It comprises 50 multi-session dialogue instances, 135 forget requests of six types, and 1,742 test questions. Each dialogue has a control version without the requests, so that only systems that have stored the target can earn credit for forgetting. Across the paired versions, five test suites measure storage, removal of the requested information, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks. On Bemula, we evaluate six mainstream memory systems under three backbone LLMs, together with the Full Context LLM on two of them. The results are negative. Across all evaluated systems and backbones, no system passes more than 1.5% of forget requests end to end, and most pass none at all. These systems either leave the requested information extractable under simple attacks or delete far more than requested. Memory unlearning therefore remains an open challenge for agent memory design. We release our data and code at https://github.com/Nboxff/bemula-benchmark.

1 INTRODUCTION

LLM-agent memory research has emphasized storing and retrieving information, while explicit user-requested forgetting remains largely unaddressed. BEMULA frames memory unlearning as deleting requested information while preserving other memories.

1 INTRODUCTION

  • Failure to forget can leave sensitive information accessible through retrieval, inference, or prompt-based extraction, while outdated memories can interfere with decisions.Redundant representations—such as raw turns, extracted facts, and compressed notes—can leave other copies intact after one is deleted.Source: §1 · p. 1
  • Existing agent-memory benchmarks mainly assess retention and reasoning, while update benchmarks do not test removal after explicit user requests.The paper defines memory unlearning as removing designated information while preserving the rest of memory.Source: §1 · p. 2
  • BEMULA pairs each dialogue containing forget requests with a control version and evaluates retention, efficacy, relearning, attacks, and preservation.The paired design lets only systems that retain the target in the control version earn credit for forgetting.Source: §1 · p. 2
  • BEMULA comprises 50 dialogue instances, 746 sessions, and 1,742 test questions, built through a five-stage multi-agent pipeline.The benchmark organizes questions into five probe suites and grades answers with an LLM judge.Source: §1 · p. 2

2 RELATED WORK

Agent-memory research progressed from long-context retrieval to multi-session and memory-augmented benchmarks, while memory methods diversified in how they store and organize information. Prior work rarely tests explicit forgetting in persistent agent memory, which BEMULA evaluates against paired controls.

2 RELATED WORK

  • Long-context benchmarks such as LongBench, RULER (Hsieh et al., 2024), and ∞-Bench test retrieval within one long input.LoCoMo and LongMemEval (Wu et al., 2024) instead probe recall across conversation sessions.Source: §2 · p. 3
  • MemoryAgentBench, MemoryArena (He et al., 2026), and MemBench evaluate memory-augmented agents across varied competencies and scenarios.These benchmarks, like earlier agent-memory evaluations, focus on retaining and using information rather than explicit forgetting.Source: §2 · p. 3
  • Existing forgetting evaluations test replacement or implicit invalidation, but not explicit requests paired against controls to separate forgetting from failure to store.LongMemEval and MemoryAgentBench test latest-fact responses; Memora does not probe forgotten content for extraction or relearning.Source: §2 · p. 3
  • Agent memory methods range from raw-turn retrieval to structured facts, temporal graphs, linked notes, and hierarchical stores, but deletion generally serves freshness rather than user intent.Machine and in-context unlearning address model parameters or single-session behavior, whereas BEMULA targets persistent external memory in multi-session agents.Source: §2 · p. 3

3 THE BEMULA BENCHMARK

BEMULA formalizes memory unlearning as deleting requested information while preserving unrelated memories and the ability to learn similar facts. It evaluates this task through paired multi-session dialogues, five question suites, and control-conditioned scoring.

3.1 TASK FORMULATION

  • After a forget request, target information must be neither retrievable nor inferable, while unrelated content remains intact and similar facts remain learnable.Each forget dialogue is paired with a control version in which requests and confirmations are replaced by a neutral interaction.Source: §3.1 · p. 4

3.2.1 PROFILE SYNTHESIS

  • Profiles pair synthetic user attributes with atomic facts, while derived targets combine at least two facts and are separately checked against their sources.Derived facts are verified to follow from their cited sources without external knowledge, speculation, or assumptions.Source: §3.2.1 · pp. 4–5

3.2.2 DIALOGUE PLANNING

  • Six request types cover factual, compositional, semantic, inferential, temporal, and replacement deletion; targets are scheduled before requests, with F4 source facts mentioned instead.For F4, the benchmark tests forgetting the derived conclusion, not whether its source facts survive.Source: §3.2.2 · p. 5

3.2.3 MULTI-AGENT DIALOGUE SIMULATION

  • A user and assistant simulator generate sessions under director and verifier supervision; planned requests are injected afterward, with neutral exchanges substituted in controls.The assistant simulator sees only the last ten turns of its current session, and an LLM verifier checks explicit deletion intent.Source: §3.2.3 · p. 5, §3.2.4 · pp. 5–6

3.2.5 QUESTION GENERATION

  • Five question suites test retention, forgetting efficacy, preservation, relearning, and attacks; attack questions use five fixed frames whose wording is preserved.Replacement requests ask for the new value in the efficacy suite, while F6 requests are exempt from relearning questions.Source: §3.2.5 · p. 6

3.3 BENCHMARK STATISTICS

  • 50 paired dialogue instances span 746 sessions and 36,900 turns per version, with 135 requests targeting 269 facts and 1,742 questions.Instances range from about 100 turns to 3,500 turns; paired versions total 73,800 turns and average 84.8K tokens.Source: §3.3 · p. 6

3.4 BENCHMARK METRICS

  • Evaluation answers each question in a fresh session with only the query in working context, and conditions forget-version efficacy, preservation, and attack scores on control probes.Retention accuracy measures control-version memorization; efficacy, preservation, relearning, and attack robustness measure their respective suites.Source: §3.4 · pp. 6–7
  • A request passes only if all its questions and the instance’s preservation questions are correct; the overall score multiplies forgetting ability by a usability guard.Because the score is a product, a zero on any of its four component metrics zeroes the total.Source: §3.4 · p. 7

4 EXPERIMENTS

BEMULA results show that end-to-end forgetting is rare, while systems struggle to balance deletion with preservation and resist extraction. Outcomes also vary by request type and backbone.

4.2 MAIN RESULTS

  • Request pass rate never exceeds 1.5% (LangMem on DeepSeek-V4.1-Flash), while Score peaks at 23.1% on GPT-5.6-Luna (Full Context), 31.0% on DeepSeek-V4.1-Flash (LangMem), and 9.5% on Qwen3-32B-AWQ (SimpleMem).Most systems have a request pass rate of zero.Source: Table 1 · p. 8, §4.2 · p. 8

4.1 EXPERIMENTAL SETUP

  • Systems ingest forget or control dialogues turn by turn, then answer test questions in a fresh session; all answers are judged by DeepSeek-V4.1-Flash.The evaluation covers six memory systems and Full Context under up to three backbones, with Qwen3-32B-AWQ omitted from two evaluations because of context limits.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

  • Mem0, LightMem, and SimpleMem retain above 87% on two backbones but have FE below 25%; LangMem reaches FE 40.6 and PR 74.0 on DeepSeek-V4.1-Flash.LangMem's higher FE on GPT-5.6-Luna comes with PR of 20.0, and its effectiveness varies by backbone.Source: §4.2 · p. 8
  • Relearning accuracy never falls below 71.1, but attack robustness stays at or below 35.7 in every setting and reaches 4.7 for Mem0 on GPT-5.6-Luna.Prompts can often extract information even when a system reports that it is gone.Source: §4.2 · pp. 8–9

4.3 RESULT ANALYSIS

  • F3 and F5 are hardest, while F6 efficacy reaches at least 61.9 across systems on the two stronger backbones even as old values survive attacks.Backbone changes also matter: Full Context FE rises from 26.8 on GPT-5.6-Luna to 53.6 on DeepSeek-V4.1-Flash.Source: Figure 4 · p. 9, §4.3 · p. 9
  • Most requests never reach storage: Naive RAG has no deletion path, offline updates miss dialogue-time requests, and SimpleMem's notes are not subsequently edited.A-MEM's note evolution can rewrite a related note when a request arrives, reaching FE 33.0 on GPT-5.6-Luna and 29.5 on DeepSeek-V4.1-Flash.Source: §4.3 · p. 9
  • Some systems over-delete, while Full Context over-refuses; on DeepSeek-V4.1-Flash it gives blanket refusals in 66 of 81 preservation failures.LangMem's deleted content on GPT-5.6-Luna can include facts not targeted by a request, such as a street address.Source: §4.3 · p. 9

5 CONCLUSION

The authors present BEMULA as a benchmark for memory unlearning and report that current systems retain information more reliably than they forget it on request. Its scope is limited to synthetic conversational data, and the authors identify tool use as a next step.

5 CONCLUSION

  • Across six memory systems and Full Context under up to three backbones, systems retain target information reliably, yet forgetting on demand largely fails and almost no request passes its full suite.The authors conclude that memory unlearning remains an open challenge.Source: §5 · p. 9
  • The study uses synthetic dialogues from one generation pipeline and covers conversational memory, not forgotten values left in tools, caches, logs, or external services.The authors identify real-user dialogues and tool use as natural extensions.Source: §5 · p. 9, ETHICS STATEMENT · p. 10
  • The released repository includes the data, code, prompts, configurations, generation pipeline, and evaluation harness; reproducing experiments requires access to LLM APIs.Runs record model, judge, and configuration metadata, and sampling uses a seeded default.Source: REPRODUCIBILITY STATEMENT · p. 10

A EVALUATION PROTOCOL AND ITEM-LEVEL METRICS

BEMULA pairs control and forget evaluations to distinguish genuine forgetting from failure to store a fact, then examines whether systems can remove requested information without losing unrelated memories. The evaluated systems’ deletion paths help explain why forgetting remains incomplete.

A EVALUATION PROTOCOL AND ITEM-LEVEL METRICS

  • Each instance has a control pass and a reset forget pass; paired probes condition item-level metrics on control success, with F4 testing suppression of an inferred conclusion.For F4, either suppressing the inference while retaining source facts or deleting both sources and conclusion is acceptable.Source: A EVALUATION PROTOCOL AND ITEM-LEVEL… · p. 13, Table 2 · p. 13

B HUMAN VALIDATION OF THE LLM JUDGE

C.1 INTEGRATION

  • A shared adapter standardizes system resets, turn-by-turn ingestion, and queries, while the same agent LLM answers from each system’s retrieved memories.Retrieval budgets follow stock configurations, and systems requiring embeddings use text-embedding-3-small.Source: §C.1 · p. 14

C.2 DELETION POLICY

  • Deletion is tested as a system’s own response to an ordinary dialogue request, not as a manual store-cleaning operation.LangMem has a removal channel enabled for evaluation, but it acts only on notes retrieved for the current batch; other systems vary in whether they can remove or rewrite stored content.Source: §C.2 · p. 14
  • Systems without a removal path retain targets that resurface under attack, retrieved-note deletion can be incomplete, and A-MEM’s overwrite path favors replacement over deletion.Replacement requests (F6) succeed almost everywhere for A-MEM, while deletion requests largely survive.Source: §C.2 · p. 15

D DETAILED BENCHMARK STATISTICS

BEMULA combines synthetic multi-session profiles with six kinds of forget requests and paired probes for retention, forgetting, preservation, relearning, and extraction attacks. Its probes test both whether a target is forgotten and whether unrelated information remains available.

D DETAILED BENCHMARK STATISTICS

  • The 50 synthetic profiles contain 2,993 atomic facts and 77 derived facts, including 1,056 facts labeled medium or high sensitivity.Profiles span 10 to 114 days of dialogue, with a mean of 49 days.Source: D DETAILED BENCHMARK STATISTICS · p. 15
  • Dialogue tiers range from 96–164 turns in short instances to 2,688–3,500 turns in xlong instances, with more requests and questions in longer instances.Short instances have one or two forget requests and 11–33 questions; xlong instances have four to six requests and 50–70 questions.Source: D DETAILED BENCHMARK STATISTICS · p. 15, Table 5 · p. 16
  • The benchmark includes 135 requests across six types, with 13 F4 requests targeting unstated derived facts, and 1,742 questions spanning five suites.The request types are factual, compositional, semantic, inferential, temporal, and replacement; 114 questions test relearning and 1,030 are attack probes.Source: D DETAILED BENCHMARK STATISTICS · pp. 15–16

E.1 ONE FORGET REQUEST OF EACH TYPE

  • Examples show requests ranging from forgetting one health fact to deleting a whole category, suppressing an inference, or replacing an old preference with a new one.Replacement probes expect the new value rather than the forgotten marker.Source: §E.1 · p. 16

E.2 PAIRED PROBES

  • Paired probes ask the same question on control and forget versions, awarding forgetting credit only when the target was first shown to be stored.A system that fails the control retention probe earns no efficacy credit for that target.Source: §E.2 · pp. 16–17

E.3 PRESERVATION AND RELEARNING PROBES

  • Preservation probes check that untargeted facts remain answerable, while relearning probes test acquisition of similar new facts after a request.Replacement requests are exempt from relearning because their new value is already assessed by the efficacy probe.Source: §E.3 · p. 17

E.4 ATTACK SUITE EXAMPLE

  • Five independently scored attack questions probe one forgotten fact through indirect, role-play, jailbreak, Boolean, and multi-hop extraction.The multi-hop question reconstructs the target from related facts that remain in memory.Source: §E.4 · p. 17, Figure 5 · p. 17

F ADDITIONAL RESULTS ON BEMULA-LITE

BEMULA-Lite offers a shorter, cost-efficient subset while retaining the full benchmark’s scoring setup. Its scores remain comparable to those on BEMULA.

F ADDITIONAL RESULTS ON BEMULA-LITE

  • BEMULA-Lite contains 31 short and medium instances with 848 questions, omitting the long and xlong tiers that dominate run cost.The omitted tiers span 916 to 3,500 turns per instance, and the longest full-benchmark dialogue reaches about 379K tokens.Source: F ADDITIONAL RESULTS ON BEMULA-LITE · p. 17
  • The subset keeps the full benchmark’s forget and control versions, question suites, and metric definitions, so its scores are computed the same way.The BEMULA-Lite results are reported under three backbone LLMs.Source: F ADDITIONAL RESULTS ON BEMULA-LITE · p. 17
  • Dropping the long and xlong tiers reduces the run cost while preserving score comparability with the full benchmark.The subset retains 15 short and 16 medium instances.Source: F ADDITIONAL RESULTS ON BEMULA-LITE · p. 17

G THE EFFICACY AND PRESERVATION TRADE-OFF

Across evaluated systems and backbones, forgetting efficacy and preservation do not jointly reach high levels. LangMem on DeepSeek-V4.1-Flash comes closest to the upper-right region, but selective forgetting remains unsolved.

G THE EFFICACY AND PRESERVATION TRADE-OFF

H ATTACK ROBUSTNESS BY ATTACK TYPE

Attack success varies by system: Full Context is vulnerable to jailbreak prefixes, while systems that forget too little are more exposed to indirect questions and Boolean guesses.

H ATTACK ROBUSTNESS BY ATTACK TYPE

  • Full Context’s attack robustness falls to 9.7 on GPT-5.6-Luna and 25.2 on DeepSeek-V4.1-Flash against jailbreak prefixes.The table reports attack-robustness percentages by attack type.Source: H ATTACK ROBUSTNESS BY ATTACK TYPE · p. 18, Table 7 · p. 19
  • Indirect questions extract Mem0’s deleted value in almost every attempt, with AR 2.4 on GPT-5.6-Luna.For systems that forget too little, indirect questions and Boolean guesses are usually the most effective attacks and require no prompt trickery.Source: H ATTACK ROBUSTNESS BY ATTACK TYPE · p. 19, Table 7 · p. 19
  • Full Context has the highest average robustness on GPT-5.6-Luna at 27.6, while LangMem leads on DeepSeek-V4.1-Flash at 35.7 but still leaks targets in roughly two thirds of attacks.Full Context’s result is carried by Boolean and multi-hop attacks.Source: H ATTACK ROBUSTNESS BY ATTACK TYPE · p. 19, Table 7 · p. 19

I SUITE METRICS BY FORGET REQUEST TYPE

Across forget-request types, retention varies less than efficacy, while relearning remains high and attack robustness remains low.

I SUITE METRICS BY FORGET REQUEST TYPE

J EFFECT OF DIALOGUE LENGTH

Longer dialogues weaken deletion and preservation for several systems, while attack robustness changes little with length.

J EFFECT OF DIALOGUE LENGTH

  • LangMem’s note-removal efficacy falls from 66.7 on the shortest bin to 21.9 on the longest for GPT-5.6-Luna.Figure 8 groups instances into five dialogue-length bins and recomputes suite metrics for each.Source: J EFFECT OF DIALOGUE LENGTH · p. 19, Figure 7 · p. 20
  • Preservation declines with dialogue length from 70.0 to 26.7 for Full Context on DeepSeek-V4.1-Flash and from 83.3 to 46.7 for Naive RAG on GPT-5.6-Luna.The reported pattern indicates growing collateral damage as dialogues lengthen.Source: J EFFECT OF DIALOGUE LENGTH · p. 19, Figure 7 · p. 20
  • Naive RAG’s relearning accuracy on GPT-5.6-Luna drops from 91.7 to 51.4 on the longest dialogues, whereas Full Context stays near 100 throughout.Full Context retains the raw history in context; store-based systems lose relearning performance on the longest dialogues.Source: J EFFECT OF DIALOGUE LENGTH · p. 20
  • Attack robustness barely changes with dialogue length, while Qwen3-32B-AWQ’s curves remain compressed toward the bottom at every length.The authors identify backbone capacity, rather than dialogue length, as the binding constraint for that backbone.Source: J EFFECT OF DIALOGUE LENGTH · p. 20, Figure 7 · p. 20

K BOOTSTRAP CONFIDENCE INTERVALS

The section estimates uncertainty with instance-level resampling, finds the central negative results robust, and documents generation and evaluation prompt conventions.

K BOOTSTRAP CONFIDENCE INTERVALS

  • The cluster bootstrap resamples 50 instance IDs 10,000 times, keeping items from both dialogue versions together to preserve control-probe conditioning.Questions within an instance share a profile and dialogue history, so the instance is the outermost independent unit.Source: K BOOTSTRAP CONFIDENCE INTERVALS · p. 20
  • On DeepSeek-V4.1-Flash, LangMem and Full Context differ by 2.5 points on Score, but their paired 95% CI of [−2.4, 7.6] is consistent with a tie.LangMem leads SimpleMem by 14.5 points, with a paired 95% CI of [9.5, 19.8].Source: K BOOTSTRAP CONFIDENCE INTERVALS · p. 20
  • No system’s efficacy interval exceeds 60.2, and no attack-robustness interval exceeds 41.4, so the negative findings do not hinge on the sampled 50 instances.Per-type and per-tier comparisons are descriptive rather than confirmatory because some cells contain few units, including 13 F4 requests and three xlong instances.Source: K BOOTSTRAP CONFIDENCE INTERVALS · p. 21

L GENERATION AND EVALUATION PROMPTS

  • The forget-request prompt requires every style to unambiguously request removal and requires replacement requests to state the new value explicitly.The evaluation judge applies separate value-gold and sentinel-gold rules, including marking leakage when a protected value is revealed partially or in paraphrase.Source: L GENERATION AND EVALUATION PROMPTS · pp. 24–25

Related papers

Loading 2610.18052v1…