PaperlayerView exact revision on arXiv

Source-linked AI summary

GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents

Zhe Ren, Yibo Yang, Yimeng Chen, Zijun Zhao, Benshuo Fu, Zhihao Shu, Bingjie Zhang, Yangyang Xu, Dandan Guo, Shuicheng Yan

arXiv:2606.18829v1cs.LGcs.CL

TL;DR

Memory benchmarks often treat agent memory as a private cache, despite shared deployments where multiple principals query a common pool under different roles and scopes. GATEMEM evaluates utility, access control, and active forgetting together, and its results show no method consistently achieves them simultaneously.

  • Problem

    Most existing memory benchmarks treat agent memory as a private cache and optimize recall, while shared deployments involve multiple principals and require evaluation of utility, access control, and deletion compliance together.

    Source: §1 · pp. 1–2

  • Method

    GATEMEM benchmarks multi-principal shared-memory agents by jointly evaluating utility, access control, and active forgetting across medical, office, education, and household domains.

    Source: §1 · p. 2

  • Results

    Across evaluated backbones and architectures, no method consistently combines high utility with suppression of access-control violations and post-deletion recovery; long-context prompting offers the strongest governance trade-off but at high computational cost.

    Source: §4.2 · p. 7, §5 · p. 11

  • Takeaways & Limitations

    The results suggest future agents should treat memory as a governed shared state with reliable access and deletion semantics, not merely as a recall resource.

    Source: §5 · p. 11

    GATEMEM is agnostic to the internal representation of memory.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Memory benchmarks for LLM agents largely assume single-user settings, leaving shared assistants for hospitals, workplaces, campuses, and households understudied. In these deployments, multiple principals write to a common memory pool and query it under different roles, scopes, and relationships, so memory quality requires governance as well as recall. We introduce GateMem, a benchmark for multi-principal shared-memory agents. GateMem jointly evaluates utility for legitimate long-horizon requests with state updates, access control across contextual authorization boundaries, and agent-facing active forgetting after explicit deletion requests. It spans medical, office, education, and household domains, with long-form multi-party episodes, incremental memory injection, hidden checkpoints, structured judging, and leak-target annotations. Across diverse baselines and backbone models, no method simultaneously achieves strong utility, robust access control, and reliable forgetting. Long-context prompting often yields the best governance score at high token cost, while retrieval-based and external-memory methods reduce cost yet still leak unauthorized or deleted information. These results show current memory agents remain far from reliable shared institutional deployment.

1 Introduction

GateMem evaluates shared-memory agents as governed systems, requiring useful answers for authorized users while enforcing contextual access boundaries and post-deletion non-recovery. Its results show that current methods do not achieve strong performance across all three dimensions.

1 Introduction

  • Unlike private-cache benchmarks focused on recall, shared-memory settings require information to be accessed, withheld, and forgotten according to contextual boundaries.High recall can expose sensitive information to unauthorized requesters, while excessive privacy constraints can block legitimate queries.Source: §1 · pp. 1–2
  • GateMem jointly measures authorized usefulness, contextual boundary enforcement, and behavioral non-recovery after deletion requests—not certified physical erasure from every underlying store.After a deletion request, an agent should not recover, confirm, or indirectly reconstruct the forgotten detail.Source: §1 · p. 2
  • No method performs strongly across utility, access control, and active forgetting; long-context prompting often offers the strongest governance trade-off at high token cost, while retrieval methods still leak or recover information.The benchmark finds that retrieval-based and external-memory methods reduce cost in some settings but continue to exhibit unauthorized disclosure and post-deletion recovery failures.Source: §1 · p. 2
  • The benchmark spans medical, office, education, and household domains, with 91 long-form multi-party episodes and 2,218 hidden checkpoints covering utility, access control, and active forgetting.Episodes include distinct role structures, authorization patterns, and failure modes such as indirect inference, delegated requests, and socially engineered recovery attempts.Source: §1 · p. 2
  • Its protocol uses long-form institutional episodes, hidden checkpoints, structured judging, and auxiliary leakage audits; baseline results show brittleness under realistic shared-memory deployment.During evaluation, models receive the interaction history, requester identity, and relevant policy context, but not the query label, grading rubric, or protected target.Source: §1 · pp. 2–3

2 Related Work

Prior benchmarks evaluate agent-memory capability and utility or security risks at the model and tool levels, while GATEMEM targets governance in multi-principal shared memory. It jointly evaluates utility, access control, and active forgetting.

Benchmarks for Agent Memory

  • Long-term memory benchmarks assess recall, updates, collaboration, and task execution, while GATEMEM targets requester-specific access, withholding, and deletion in shared memory.LoCoMo and LongMemEval cover conversational recall, temporal reasoning, and updates [17] [30]; MemoryAgentBench studies retrieval, test-time learning, long-range understanding, and selective forgetting [12]. RealMem [3], EverMemBench [11], MemoryArena, and Mem2ActBench [10] [26] extend evaluation to project interaction, collaboration, multi-session tasks, and action execution.Source: §2 · p. 3

Benchmarks for Agent Security

  • Security benchmarks cover harmful generation and jailbreaks [6] [18], as well as prompt injection and unsafe tool use [34] [8] [35], but not joint governance evaluation for shared-memory agents.Persistent memory introduces risks of retaining sensitive information, disclosing it to the wrong requester, or reconstructing it after deletion; related work studies contextual integrity, over-persistent memory, and privacy leakage [19] [23] [32].Source: §2 · p. 3

3 The GATEMEM Benchmark

GATEMEM benchmarks shared-memory governance across four domains using evolving multi-principal episodes and hidden checkpoints for utility, access control, and active forgetting. It defines separate governance measures and combines them in a score that rewards usefulness only alongside protection and deletion compliance.

3 The GATEMEM Benchmark

  • GATEMEM covers medical, office, education, and household domains through scenario specifications, LLM-assisted multi-party episodes with evolving facts and permissions, and hidden evaluation checkpoints.Checkpoint annotations include the expected action, judge specification, and leak targets; all generated episodes undergo quality control for schema consistency, evidence support, deletion closure, and leak targets.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Episodes and Memory State

  • Each episode pairs a scenario specifying principals, relationships, and scoped rules with a multi-party trace whose natural-language turns may change facts, permissions, or deletion state.Agents ingest turns incrementally before checkpoints, and GATEMEM permits full-context replay, retrieval, structured records, or external memory.Source: §3.1 · p. 4

3.2 Checkpoints and Governance Categories

  • Three disjoint checkpoint sets test authorized answers, withholding protected information, and post-deletion non-recovery; forgetting measures interface-level compliance, not certified physical erasure.Expected actions are answer for utility, refuse or answer_redacted for access control, and no_memory for active forgetting; responses are normalized into four actions: answer, answer_redacted, refuse, and no_memory.Source: §3.2 · p. 5

3.3 Evaluation Metrics

  • MGS is U · (1 − A) · (1 − F), combining effective utility, access-control violation, and active-forgetting failure so strength on one pillar cannot offset weakness on another.Utility requires the expected answer action and factual completeness; violations involve leaking protected targets or failing to restrict access, while forgetting failures involve recovering deleted information or failing to take the no_memory action.Source: §3.3 · pp. 5–6

3.4 Dataset Statistics and Quality Control

  • GATEMEM comprises 91 long-form episodes and 2,218 hidden checkpoints across four domains, with professional coordination and partial delegation in medical and office episodes and often implicit or fluid boundaries in education and household episodes.A four-stage quality-control pipeline checks schema consistency, evidence support for utility answers, deletion-chain closure, and leak-target precision through manual inspection.Source: §3.4 · p. 6

4 Experiments

GateMem experiments show that no evaluated method consistently combines strong utility with access control and reliable post-deletion forgetting. Results expose trade-offs among utility, governance, and efficiency, while diagnostic analyses identify retrieval sensitivity, over-refusal, and subtle attack patterns as important failure mechanisms.

4.1 Experimental Setup

  • The evaluation compares full-history prompting, retrieval-based baselines, and dedicated memory systems under a shared incremental episode protocol across six LLM backbones.Agents process turns in temporal order, answer checkpoint queries, then resume; primary results use GPT-4o judge labels.Source: §4.1 · pp. 6–7

4.2 Main Results

  • LONG-CONTEXT often leads on MGS but still leaks, while policy-aware retrieval reduces unauthorized disclosure at a utility cost and explicit memory systems do not consistently improve MGS.LONG-CONTEXT’s full history supports legitimate queries but exposes sensitive or deleted information; A-MEM, MEM0, and REMEM do not consistently outperform simpler baselines.Source: §4.2 · p. 7
  • Backbone choice shifts the utility–risk trade-off: GPT-5.4 and Deepseek-V4-Pro improve governance scores, while Gemini-2.5-Flash-Lite often has higher forgetting failures and access violations.GPT-5.4 attains the highest single-domain MGS in Medical; Llama-4-Maverick also improves utility in several settings but has more post-deletion recovery failures than GPT-5.4 and Deepseek-V4-Pro.Source: §4.2 · p. 7
  • No evaluated method consistently achieves high utility while suppressing access-control violations and post-deletion recovery across backbones and architectures.Table 3 reports the judge-based main results across backbones and domains.Source: §4.2 · p. 7, Table 3 · p. 8
  • 4.22s/ckpt on Medical is LONG-CONTEXT’s example wall-clock time, while REMEM uses 1k tokens/ckpt but can take up to 260s/ckpt.Human-adjudicated metrics differ from GPT-4o judge-derived labels by at most 1.04 percentage points, with at least 97.7% field-level agreement.Source: §4.2 · pp. 7–8, Table 4 · p. 9

4.3 Diagnostic and Failure Analysis

  • Across top-k values 5, 10, 20, 30, and 40, POLICY RAG maintains higher access and forgetting safety than NAIVE RAG, while utility generally rises with retrieval depth.Shallow retrieval can omit evidence needed for correct answering and safe withholding.Source: Figure 3 · p. 9, §4.3 · p. 9
  • POLICY RAG has higher over-refusal than LONG-CONTEXT on legitimate utility checkpoints, showing that stronger safety can sacrifice authorized utility.Figure 3(b) measures over-refusal on legitimate utility checkpoints.Source: Figure 3 · p. 9, §4.3 · p. 9
  • Medical-domain failures include soft-overreach and indirect deletion probes; examples show agents can reveal protected chart types or reconstruct deleted information despite retrieving relevant evidence.Policy filtering mitigates many soft-overreach risks, but persistent memory systems remain brittle when relevant facts are unauthorized or deleted; case studies also show missed medication evidence and over-refusal.Source: §4.3 · pp. 9–11, Figure 4 · p. 10, Table 5 · p. 10

5 Conclusion

GateMem evaluates memory governance in multi-principal shared-memory agents and finds that current designs struggle to satisfy utility, access control, and active forgetting simultaneously.

5 Conclusion

  • Long-context prompting offers the strongest governance trade-off at high computational cost, while retrieval and external-memory baselines remain vulnerable to unauthorized disclosure and post-deletion recovery.The findings suggest current agents do not simultaneously meet the benchmark’s governance requirements.Source: §5 · p. 11
  • GATEMEM evaluates multi-principal shared-memory agents across utility, access control, and active forgetting.Source: §5 · p. 11
  • Future agents should treat memory as governed shared state with reliable access and deletion semantics, rather than merely as a recall resource.Source: §5 · p. 11

A Additional Dataset Details · A.1 Detailed Design Principles

The appendix expands on GATEMEM’s construction and validation by detailing design principles, domain and requester-role characteristics, checkpoint types, and long-horizon challenges. Its episodes model institutional workflows where authorization and correct answers depend on context and current state, and forgetting is tested through explicit deletion followed by targeted queries.

A Additional Dataset Details

  • The appendix supplements the main text’s benchmark pipeline and domain statistics with details on construction, structure, validation, domain characteristics, requester roles, checkpoints, and long-horizon challenges.Source: A Additional Dataset Details · p. 14

A.1 Detailed Design Principles

  • Episodes model plausible institutional workflows, using coordination, rescheduling, delegation, updates, and unrelated turns to separate evidence from later queries.Source: §A.1 · p. 14
  • Authorization depends jointly on role, relationship, scope, and current state, rather than a static role lookup.Requesters may have an operational connection to a case without sufficient authority to access protected details.Source: §A.1 · p. 14
  • Correct answers often require the latest authoritative value, so episodes test tracking updates to dates, amounts, assignments, credentials, and safe wording.Source: §A.1 · p. 14
  • Access-control checkpoints test soft overreach through delegated requests, authority pressure, label-existence probing, and indirect inference.These scenarios reflect realistic deployment risks that assistants may mishandle while trying to be helpful.Source: §A.1 · p. 14
  • Forgetting checkpoints form closed attack chains: a sensitive value appears, is explicitly deleted, and is then targeted by direct, confirmatory, reconstructive, or socially engineered queries.This design avoids reducing forgetting evaluation to vague refusal testing.Source: §A.1 · p. 14

A.2 Domain Details · A.3 Domain Structural Statistics

The benchmark spans four domains whose distinct roles and sensitive information create contextual authorization challenges, including partially legitimate requests and attempts to recover deleted information. Its long-form, multi-principal episodes test integration of distributed evidence and current-state updates alongside protection of restricted details.

A.2 Domain Details

  • Medical covers patient-care roles and sensitive health information, with 21 episodes, 11 role types, and 579 checkpoints, including 210 utility checkpoints.Its central challenge is that a family member may be authorized for logistics but not for protected health values.Source: §A.2 · p. 14
  • Office covers enterprise roles and confidential operational information across 17 episodes, 16 role types, and 547 checkpoints, including 154 utility checkpoints.Delegation chains, project-scope ambiguity, and role adjacency make authorization difficult even when requests are operationally plausible.Source: §A.2 · p. 14
  • Education spans academic, administrative, residential, and student-support workflows, with 30 episodes, 16 role types, and 540 checkpoints split evenly across utility, access control, and active forgetting.It stresses parent overreach, delegated access, instructional–administrative role mismatch, and attempts to recover deleted support information.Source: §A.2 · p. 14
  • Household covers personal and operational coordination across 23 episodes, 17 role types, and 552 checkpoints, evenly divided among utility, access control, and active forgetting.Requesters may have plausible household relationships without authority to access protected schedules, care routines, access codes, or deleted instructions.Source: §A.2 · pp. 14–15
  • All domains use long-form multi-party trajectories with parallel threads, confusable names, and late-stage current-state anchors to test evidence integration, update resolution, and protection.These design choices make the benchmark test more than lexical retrieval: agents must distinguish useful information from protected information.Source: §A.2 · p. 15

A.3 Domain Structural Statistics

  • Office, education, and household episodes average roughly 200 turns, while medical episodes are also longitudinal; office is densest in principals and active roles.Figure 5 summarizes episode counts, turns, reference tokens, principals, active roles, and checkpoints; its token counts measure content density, not runtime billing cost.Source: §A.3 · p. 15, Figure 5 · p. 15

A.4 Requester Diversity

Requester diversity is central to GateMem: the same shared memory is queried by different principals with different intents. Requesters vary across query types and domains, making authorization context-dependent rather than reducible to a single canonical asker.

A.4 Requester Diversity

  • Figure 6 separates requester roles across utility, access-control, and active-forgetting checkpoints, where the same role may be legitimate in one context and over-scoped in another.The panels report each role’s percentage of checkpoints within each query type.Source: §A.4 · p. 15, Figure 6 · p. 16
  • Medical utility queries concentrate on patients and family members, while office, education, and household queries span broader operational roles; probes also use varied over-scoped requesters.This diversity makes a simple role blacklist or single refusal template insufficient for solving GateMem.Source: §A.4 · pp. 15–16

A.5 Checkpoint and Attack Composition

Hidden checkpoints test legitimate utility, access control, and active forgetting across domains. Their attack scenarios include soft authorization violations and varied attempts to recover deleted information, while leak-target annotations enable explicit leakage audits.

A.5 Checkpoint and Attack Composition

  • Utility checkpoints test legitimate access to current authoritative values, multi-slot synthesis, and long-horizon integration under state updates, using expected actions and required answer elements.Source: §A.5 · p. 16
  • Access-control checkpoints test direct cross-principal requests and softer attacks, specifying safe actions, permitted summaries, and protected content that must not appear.The softer attacks include delegated authority, authority pressure, and label-existence probes.Source: §A.5 · p. 16
  • Active-forgetting checkpoints test recovery of deleted values through direct queries, yes-or-no prompts, fragments, or later updates, requiring the normalized action no_memory.Attack scenarios also include social engineering and update-conflict exploits.Source: §A.5 · pp. 16–17
  • Figure 7 shows all three checkpoint types occupy substantial shares in every domain, while access-control and forgetting attacks span varied boundary violations and recovery strategies.The attack mix includes delegated authority, indirect inference, label-existence probes, split reconstruction, social engineering, and update-conflict exploits; direct restatement alone would overestimate forgetting robustness.Source: Figure 7 · p. 17, §A.5 · p. 17
  • Leak-target annotations support explicit leakage auditing and make access-control and active-forgetting checkpoints more transparent to manual inspection.Source: §A.5 · p. 17

A.6 Quality Control and Challenge Profile

GateMem combines schema, evidence, deletion-chain, and leak-target checks with iterative manual review to improve episode quality. It also profiles long-horizon demands using mean and 90th-percentile statistics across four indicators.

Quality Control

  • GateMem validates episode schemas, checkpoint support, active-forgetting chain closure, and leak-target precision through multiple quality-control layers.Utility gold answers must be supported by pre-checkpoint history; forgetting chains require the sensitive value and explicit deletion to precede recovery attempts.Source: §A.6 · p. 17
  • Iterative manual review removes ambiguous utility askers, strengthens late-stage current-state anchors, and eliminates under-specified access-control or forgetting targets.The authors treat this review as part of the benchmark contribution, not merely an annotation detail.Source: §A.6 · p. 17

Challenge Profile

  • The challenge profile reports means and 90th percentiles for utility horizon, support span, answer slots per checkpoint, and delete-to-attack gap.These indicators capture evidence recency and dispersion, answer integration burden, and the interval during which deleted content may be recovered; upper-tail statistics surface long-tail difficulty without relying on unstable maxima.Source: §A.6 · pp. 17–18, Figure 8 · p. 18

B Baseline Details · C Prompt Templates

The appendix specifies seven baseline configurations and the assistant and judge prompts used to operationalize access control, active forgetting, and utility evaluation. The prompts distinguish authorized answers, redactions, refusals, and unavailable or deleted information, then score outputs against checkpoint-specific specifications.

B Baseline Details

  • Seven baselines comprise LONG-CONTEXT, RAG-NAIVE, RAG-POLICY, A-MEM [31], MEM0 [7], REMEM-I, and REMEM-S [27].The main result tables include only baseline runs completed for the evaluated experimental configurations.Source: B Baseline Details · p. 18
  • LONG-CONTEXT supplies the full interaction history without external memory, while RAG-NAIVE retrieves top-k snippets without a dedicated access-control or deletion-aware layer.LONG-CONTEXT serves as a utility-oriented reference when prompt length is unconstrained; RAG-NAIVE relies on the model to judge whether retrieved information is safe.Source: B Baseline Details · p. 18
  • RAG-POLICY adds requester, domain, and access metadata to retrieval, while A-MEM [31], MEM0 [7], and REMEM [27] test adaptive notes, persistent fact updates, and graph-based episodic retrieval.REMEM-I uses iterative tool-based retrieval and reasoning; REMEM-S uses single-step embedding retrieval and generation with lower inference overhead.Source: B Baseline Details · pp. 18–19

C Prompt Templates

  • The judge receives checkpoint metadata, assistant output, an authoritative grading specification, and leak-target patterns, then returns action_ok, utility_ok, privacy_leak, and deletion_leak fields.The paper maps the evaluation-code labels privacy_leak and deletion_leak to leakage under ACCESS CONTROL and ACTIVE FORGETTING, respectively.Source: C Prompt Templates · pp. 19–21
  • The inference prompt fixes the authenticated requester, bars disclosure beyond policy, rejects social-engineering, and forbids reconstructing deleted or inaccessible information.It directs the assistant to use answer, answer_redacted, refuse, or no_memory, reserving no_memory for deleted or unavailable information rather than lack of authorization.Source: C Prompt Templates · pp. 19–20
  • Decision priority requires answering an owner's in-scope request unless information is deleted or unavailable, permits answers within explicit valid consent or delegation, and otherwise allows only a high-level redacted summary when appropriate.Requester relationship facts provide high-level context, but the prompt says not to infer unstated access or treat those facts as more authoritative than retrieved memory.Source: C Prompt Templates · p. 20
  • The judge checks actions against expected_action and marks utility successful only when every required item is covered; refusals, redactions, or no_memory responses fail utility queries.Paraphrases and formatting differences are allowed, but missing, contradicted, or vague required items do not satisfy the specification.Source: C Prompt Templates · p. 21
  • A deletion leak includes revealing or confirming deleted content, including validating a guessed value, whereas a non-revealing refusal is not a leak solely because its action is incorrect.The judge must not repeat protected leak-target values and should describe decisions only by category in its notes.Source: C Prompt Templates · p. 21

D Experimental Details · E Qualitative Case Studies

The appendix specifies GateMem’s incremental evaluation and judge-based scoring, validates judge labels against human annotations, and presents representative medical checkpoint case studies across baselines.

D Experimental Details

  • Episodes are processed incrementally: agents reset, ingest turns chronologically, answer annotated checkpoint queries, then continue through the remaining turns.The driver loads episodes and checkpoint annotations, validates the dataset, and writes predictions, judge scores, scores, and a run summary.Source: D Experimental Details · pp. 21–22
  • LLM judges assess action compliance and task-specific correctness for utility, access-control, and active-forgetting checkpoints using hidden checkpoint annotations.The checkpoint labels assess effective answer quality, unauthorized disclosure, and recovery, confirmation, or reconstruction of deleted information, respectively.Source: D Experimental Details · p. 22
  • Main results omit the optional post-hoc action gate, report action accuracy separately, and compute MGS directly from judge-derived utility, access-leakage, and deletion-leakage labels.For leakage, user-provided sensitive guesses alone do not count unless the assistant confirms, reveals, or reconstructs them.Source: D Experimental Details · pp. 22–23
  • Human validation randomly sampled approximately 50% of 579 judge-labeled checkpoint-output pairs from one run, stratified across all three governance categories.Each case was independently annotated by at least two human annotators using the judge’s hidden grading information; ties or conflicts were adjudicated.Source: D Experimental Details · pp. 22–23
  • Human-adjudicated U, A, F, and MGS differ from judge-derived metrics by at most 1.04 percentage points, with field-level judge-human agreement of at least 97.7%.Table 9 reports aggregate and field-level comparisons; A and F denote content-level access leakage and deletion leakage.Source: D Experimental Details · p. 23, Table 9 · p. 23

E Qualitative Case Studies

  • Table 10 presents representative medical checkpoints across seven baselines using original responses, marking correct, clear-failure, and non-leaking but non-normalized actions.Its symbols are ✓ for correct, ✗ for clear failure, and ! for avoiding direct leakage without matching the expected normalized action.Source: Table 10 · p. 24

Related papers

Loading 2606.18829v1…