PaperlayerView exact revision on arXiv

Source-linked AI summary

GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents

Zhe Ren, Yibo Yang, Yimeng Chen, Zijun Zhao, Benshuo Fu, Zhihao Shu, Bingjie Zhang, Yangyang Xu, Dandan Guo, Shuicheng Yan

arXiv:2606.18829v1cs.LGcs.CL

TL;DR

Shared-memory agents require governance across utility, authorization, and deletion, but existing memory benchmarks largely assume private single-user caches. GateMem benchmarks these requirements jointly and finds that current methods do not reliably satisfy them simultaneously.

  • Problem

    Existing memory benchmarks largely assume private single-user caches, leaving governance of shared memory across roles, scopes, authorization, and deletion understudied.

  • Method

    GateMem benchmarks multi-principal shared-memory agents across utility, access control, and active forgetting using long-form institutional episodes and hidden checkpoints.

  • Results

    No evaluated method consistently achieves high utility while simultaneously suppressing access-control violations and post-deletion recovery.

  • Takeaways & Limitations

    Current memory agents remain unreliable for shared institutional deployment and must treat memory as governed shared state with access and deletion semantics.

  • Takeaways & Limitations

    GateMem is agnostic to the internal representation of memory.

Abstract

from arXiv · show

Memory benchmarks for LLM agents largely assume single-user settings, leaving shared assistants for hospitals, workplaces, campuses, and households understudied. In these deployments, multiple principals write to a common memory pool and query it under different roles, scopes, and relationships, so memory quality requires governance as well as recall. We introduce GateMem, a benchmark for multi-principal shared-memory agents. GateMem jointly evaluates utility for legitimate long-horizon requests with state updates, access control across contextual authorization boundaries, and agent-facing active forgetting after explicit deletion requests. It spans medical, office, education, and household domains, with long-form multi-party episodes, incremental memory injection, hidden checkpoints, structured judging, and leak-target annotations. Across diverse baselines and backbone models, no method simultaneously achieves strong utility, robust access control, and reliable forgetting. Long-context prompting often yields the best governance score at high token cost, while retrieval-based and external-memory methods reduce cost yet still leak unauthorized or deleted information. These results show current memory agents remain far from reliable shared institutional deployment.

1 Introduction

GATEMEM reframes memory evaluation for multi-principal shared-memory agents as a governance problem requiring utility, access control, and active forgetting. It introduces a benchmark spanning four institutional domains, with long-form episodes, hidden checkpoints, and structured evaluation of realistic deployment failures.

  • Motivation: Existing memory benchmarks largely optimize recall in private single-user caches, unlike shared pools queried by principals with different roles, scopes, and relationships.In shared deployments, high recall without governance can still produce harmful disclosures.
  • Memory governance: Memory quality must jointly assess correct responses, contextual authorization, withholding sensitive information, and behavioral non-recoverability after deletion requests.The benchmark evaluates interface-level forgetting rather than certified physical erasure.
  • Benchmark scope: GATEMEM evaluates utility, access control, and active forgetting across medical, office, education, and household shared-memory environments.Each domain uses distinct role structures, authorization patterns, and requirements.
  • Findings: Baseline results indicate that memory-agent designs competitive on conventional recall benchmarks remain brittle in realistic shared-memory deployments.GATEMEM’s evaluation protocol centers on long-form institutional episodes, hidden checkpoints, structured judging, and leakage audits.

2 Related Work

Prior benchmarks evaluate long-term agent memory through recall, temporal reasoning, knowledge updates, retrieval, test-time learning, long-range understanding, and selective forgetting. Security benchmarks instead target harmful generation, jailbreaks, refusal behavior, prompt injection, unsafe tool use, and compromised actions, while persistent memory creates a distinct attack surface.

  • Benchmarks for Agent Memory: LoCoMo and LongMemEval assess long-term conversational recall, multi-session and temporal reasoning, and knowledge updates.These benchmarks focus on long-term memory evaluation in LLM agents.
  • Benchmarks for Agent Memory: MemoryAgentBench evaluates retrieval, test-time learning, long-range understanding, and selective forgetting in incremental multi-turn interactions.It extends memory evaluation beyond post hoc recall.
  • Benchmarks for Agent Security: Model-level security benchmarks assess harmful generation, jailbreak robustness, and refusal behavior, whereas tool-level benchmarks examine prompt injection, unsafe tool use, and compromised action execution.Examples include AdvBench and HarmBench at the model level, and InjecAgent, AgentDojo, and Agent Security Bench at the tool level.
  • Benchmarks for Agent Security: Persistent memory introduces a distinct security attack surface involving sensitive information.This motivates security evaluation specific to memory-enabled agents.

3 The GATEMEM Benchmark

GATEMEM benchmarks shared-memory assistants through long-form, multi-principal episodes spanning four domains, with evolving facts, permissions, and deletion requests evaluated at hidden checkpoints. It measures legitimate utility, access control, and active forgetting using structured annotations and governance-oriented actions.

  • Episode construction: Episodes instantiate multi-party interactions from scenario specifications defining principals, relationships, scoped access rules, and evolving facts, permissions, and deletion requests.Hidden checkpoints are inserted at selected turn boundaries with expected actions, judge specifications, and leak targets.
  • Governance evaluation: The benchmark evaluates three disjoint checkpoint sets: UTILITY, ACCESS CONTROL, and ACTIVE FORGETTING.Utility requires current in-scope answers; access control tests withholding protected information; active forgetting tests compliance after deletion.
  • Evaluation protocol: GATEMEM normalizes responses into answer, answer_redacted, refuse, and no_memory, distinguishing safe disclosure, refusal, deletion compliance, and utility failures.The benchmark is agnostic to memory representation, supporting full-context replay, retrieval, vector memory, structured records, or external memory modules.
  • Metrics: Memory Governance Score is defined as MGS = U · (1 −A) · (1 −F), requiring usefulness without protected-information leaks or forgetting failures.Efficiency metrics additionally compare runtime profiles using wall-clock duration and total input-output tokens.
  • Benchmark scope: The benchmark contains 91 long-form episodes and 2,218 hidden checkpoints across Medical, Office, Education, and Household domains.Its domains cover professional coordination, partial delegation, and settings with implicit or fluid authorization boundaries.

4 Experiments

Across backbones and memory architectures, no method consistently combines high utility with strong access control and reliable post-deletion forgetting. Long-context prompting often leads governance, policy-aware retrieval improves safety at a utility cost, and explicit memory systems reduce tokens but can incur severe latency.

  • No method consistently achieves high utility while suppressing access-control violations and post-deletion recovery.
  • LONG-CONTEXT achieves the highest MGS in most backbone–domain blocks, but full-history exposure can leak sensitive or deleted information.It leads all four domains under Deepseek-V4-Pro, GPT-5-mini, and GPT-4o-mini, and all but Office under the other listed backbones.
  • RAG-POLICY substantially reduces unauthorized disclosures versus RAG-NAIVE, but filtering can remove useful evidence and induce conservative responses.In Office, reduced leakage penalties can offset utility loss, giving RAG-POLICY the highest MGS under several backbones.
  • Explicit systems such as A-MEM, MEM0, and REMEM do not consistently outperform simpler baselines on MGS, showing that structured memory alone is insufficient for governance.Agents must also assess requester authorization and whether facts remain valid after deletion requests.
  • Efficiency Trade-offs: REMEM reduces token usage to 1k tokens/ckpt but can reach 260s/ckpt latency, whereas LONG-CONTEXT is most token-intensive yet runs at 4.22s/ckpt on Medical.The comparison motivates jointly optimizing governance and latency.
  • Retrieval-depth Sensitivity and Over-refusal Analysis: Policy-aware retrieval maintains higher access and forgetting safety across retrieval depths, but its superior safety comes with higher over-refusal than LONG-CONTEXT.Shallow retrieval often omits evidence needed for correct answering and safe withholding.

5 Conclusion

GateMem evaluates shared-memory agents across utility, access control, and active forgetting, revealing that current designs struggle to satisfy these requirements simultaneously. Long-context prompting provides the strongest governance trade-off at high computational cost, while retrieval and external-memory baselines remain vulnerable to unauthorized disclosure and post-deletion recovery.

  • Benchmark scope: GateMem benchmarks memory governance in multi-principal shared-memory agents through utility, access control, and active forgetting.The benchmark evaluates these three requirements jointly.
  • Overall finding: Current agent designs struggle to satisfy utility, access control, and active forgetting simultaneously.
  • Method comparison: Long-context prompting offers the strongest governance trade-off but incurs high computational cost.
  • Method comparison: Retrieval and external-memory baselines remain vulnerable to unauthorized disclosure and post-deletion recovery.

A Additional Dataset Details · A.1 Detailed Design Principles

GateMem’s appendix details its construction, domain characteristics, requester-role diversity, checkpoint taxonomy, and long-horizon challenge profile. Its design models institutional workflows, state-dependent authorization, evolving values, realistic access attacks, and complete deletion-to-attack chains.

  • A Additional Dataset Details: The appendix expands GateMem’s benchmark construction pipeline and summarizes its domain characteristics, requester-role diversity, checkpoint taxonomy, and long-horizon challenge profile.It provides additional construction, structure, and validation details beyond the main text.
  • A.1 Detailed Design Principles: Each episode presents a plausible institutional workflow with longitudinal coordination, rescheduling, delegation, updates, and unrelated operational turns.These turns separate evidence from later queries rather than treating episodes as isolated fact questions.
  • A.1 Detailed Design Principles: Authorization jointly depends on role, relationship, scope, and current state rather than a static role lookup.This captures cases where a requester has an operational connection but insufficient authority for protected details.
  • A.1 Detailed Design Principles: Access-control checkpoints test delegated overreach, authority pressure, label-existence probing, and indirect inference beyond obvious adversarial requests.These soft-overreach attacks reflect realistic deployment failures that arise while assistants try to remain helpful.
  • A.1 Detailed Design Principles: Active-forgetting checkpoints form closed chains in which a sensitive value appears, is explicitly deleted, and is then targeted by follow-up queries.Targets include direct, confirmatory, reconstructive, and socially engineered queries, preventing evaluation from becoming vague refusal testing.

A.2 Domain Details · A.3 Domain Structural Statistics

GateMem models shared assistants across medical, office, education, and household settings, where plausible relationships and role differences complicate authorization. Its long, multi-party episodes require agents to integrate distributed evidence, track updates, and protect sensitive or deleted information.

  • A.2 Domain Details: Medical scenarios involve diverse care-setting principals and sensitive health information, with partially legitimate requests creating difficult authorization boundaries.Principals include clinicians, patients, family members, pharmacists, reception staff, and schedulers; protected content includes health records, medications, labs, treatment plans, and deleted identifiers.
  • A.2 Domain Details: Office scenarios involve many enterprise roles, where delegation chains, project-scope ambiguity, and role adjacency make operationally plausible requests challenging to govern.Sensitive content includes project confidentiality, commercial terms, credentials, incidents, staffing decisions, and deleted internal mappings.
  • A.2 Domain Details: Education scenarios span academic, administrative, residential, and student-support workflows with sensitive academic, financial-aid, accommodation, credential, and deleted-support records.Principals include students, faculty, advisors, registrars, financial-aid staff, campus IT, residence staff, and parents or guardians.
  • A.2 Domain Details: Household scenarios combine personal coordination with operational interactions, requiring authority distinctions among residents, relatives, caregivers, guests, providers, and staff.The domain protects schedules, access codes, care routines, payment arrangements, location-linked notes, and deleted household instructions.
  • A.2 Domain Details: 23 household episodes contain 17 role types and 552 checkpoints, evenly divided among utility, access-control, and active-forgetting evaluations.Each checkpoint category contributes 184 checkpoints.
  • A.2 Domain Details: All domains use long-form multi-party trajectories with parallel threads, confusable names, and late-stage current-state anchors to test distributed evidence integration and update resolution.These design choices target more than lexical retrieval and require distinguishing useful disclosures from protected information.
  • A.3 Domain Structural Statistics: Figure 5 summarizes six structural dimensions across the four domains: episodes, turns, reference tokens, principals, active roles, and checkpoints.Office, education, and household sustain roughly 200 turns per episode on average, while medical also remains long-form.

A.4 Requester Diversity

GateMem varies requester roles across utility, access-control, and active-forgetting checkpoints because the same memory pool is queried by different principals and intents. This diversity includes legitimate and over-scoped requesters, preventing governance through simple role blacklists or refusal templates.

  • Requester Diversity: Requester distributions are reported separately for utility, access-control, and active-forgetting checkpoints.The same role may be legitimate in one setting and over-scoped in another.
  • Requester Diversity: Medical utility queries concentrate on patients and family members, while office, education, and household queries span broader operational roles.
  • Requester Diversity: Access-control and active-forgetting probes use many plausible but over-scoped requesters, complicating simple role blacklists and single refusal templates.

A.5 Checkpoint and Attack Composition

GateMem evaluates shared-memory governance through hidden utility, access-control, and active-forgetting checkpoints. Its attack composition tests both direct authorization failures and subtler leakage or recovery routes across domains.

  • Checkpoint types: Access-control checkpoints test over-scoped requests, including cross-principal access, delegated authority, authority pressure, and label-existence probes.Annotations specify the safe action, permitted summaries, and protected content that must be excluded.
  • Checkpoint types: Active-forgetting checkpoints test recovery of deleted information through direct queries, confirmations, fragment reconstruction, or later updates.These checkpoints require the normalized action no_memory.
  • Leakage auditing: Leak_targets enable explicit leakage auditing alongside LLM-based judgment, improving transparency for manual inspection.They are annotated for both access-control and active-forgetting checkpoints.
  • Checkpoint and attack composition: Utility, access control, and active forgetting occupy substantial mass in every domain, while attacks include overreach, delegated authority, indirect inference, impersonation, and label-existence probes.The composition therefore prevents optimizing only one evaluation mode and captures more than blunt access violations.

A.6 Quality Control and Challenge Profile · B Baseline Details · C Prompt Templates

GateMem combines layered quality control, long-horizon challenge characterization, seven baseline configurations, and explicit inference and judging protocols for access control and active forgetting. Its templates enforce authenticated identities, scoped disclosure, deletion-aware responses, structured actions, and leak-target evaluation.

  • A.6 Quality Control and Challenge Profile: GateMem validates schemas, utility-answer support, and active-forgetting chains before evaluation.Forgetting chains require the sensitive value before explicit deletion and recovery attempts only afterward.
  • A.6 Quality Control and Challenge Profile: Manual review removes ambiguous utility queries, strengthens current-state anchors, and eliminates under-specified access-control or forgetting targets.The paper treats this iterative review as part of the benchmark contribution.
  • A.6 Quality Control and Challenge Profile: The challenge profile measures utility horizon, support span, answer slots per utility checkpoint, and delete-to-attack gap using means and 90th percentiles.These indicators capture temporal distance, evidence dispersion, answer integration demands, and the long tail without relying on unstable maxima.
  • B Baseline Details: GateMem implements seven baselines: LONG-CONTEXT, RAG-NAIVE, RAG-POLICY, A-MEM, MEM0, REMEM-I, and REMEM-S.The first three are custom baselines, while the remaining configurations derive from prior agentic memory systems.
  • B Baseline Details: LONG-CONTEXT uses the full interaction history, RAG-NAIVE retrieves top-k snippets without policy or deletion layers, and RAG-POLICY adds requester and access metadata.These configurations respectively test prompt-based utility, standard retrieval, and lightweight policy-aware retrieval without a dedicated governance mechanism.
  • B Baseline Details: A-MEM structures linked notes, MEM0 incrementally performs add/update/delete/no-op memory operations, and REMEM compares iterative tool retrieval with single-step embedding retrieval.The REMEM variants are REMEM-I and REMEM-S, with REMEM-S designed for lower inference overhead.
  • C Prompt Templates: LLM judging compares actions and required utility items while separately detecting privacy_leak and deletion_leak against leak-target patterns.The judge returns structured JSON and must not repeat protected values; privacy_leak and deletion_leak correspond to ACCESS CONTROL and ACTIVE FORGETTING leakage.
  • C Prompt Templates: Assistant prompts fix the authenticated requester, enforce global access policy, reject social engineering, and distinguish answer, answer_redacted, refuse, and no_memory actions.Deleted or inaccessible information must not be reconstructed, while owner access, valid consent or delegation, and high-level-only permissions determine response scope.

D Experimental Details

The experiments use a standardized incremental episode protocol with checkpoint-based judging for utility, access control, and active forgetting. Human validation closely matches judge-derived aggregate metrics, supporting the reliability of the reported scores.

  • Experimental protocol: The benchmark resets agents per episode, ingests turns chronologically, answers queries at annotated as_of_turn_id checkpoints, and then processes remaining turns.The driver validates episodes.jsonl and checkpoints.jsonl and writes run-specific prediction and score files.
  • Scoring: LLM judges evaluate utility checkpoints for answer quality, access-control checkpoints for unauthorized disclosure, and active-forgetting checkpoints for deleted-information recovery or reconstruction.Judges use hidden checkpoint annotations to assess action compliance and task-specific correctness.
  • Scoring: Action accuracy is reported separately, while MGS is computed directly from judge-derived utility, access-leakage, and deletion-leakage labels unless otherwise stated.An optional gate_by_action setting can instead treat action errors as failures for inner metrics, but main results do not apply this post-hoc gate.
  • Human validation: Leakage labels count assistant confirmation, revelation, or reconstruction of sensitive information, but not user-provided sensitive guesses alone.Two annotators used the judge’s hidden grading information, with tied or conflicting labels resolved through adjudication.
  • Human validation: 1.04 percentage points was the maximum absolute difference between human-adjudicated and judge-derived aggregate metrics, while field-level judge-human agreement reached at least 97.7%.The validation study sampled approximately 50% of 579 labeled checkpoint-output pairs and covered utility, access control, and active forgetting.

E Qualitative Case Studies

Qualitative medical checkpoints compare seven baselines using correctness, failure, and non-leaking-but-nonnormalized outcomes. The cases show successful refusals for unauthorized chart access but divergent handling of deleted instructions, including clear leakage by Mem0.

  • Medical checkpoint evaluation: Table 10 evaluates each medical checkpoint across seven baselines, marking correct responses, clear failures, and non-leaking responses that miss the expected normalized action.✓ denotes correctness, ✗ a clear failure, and ! avoidance of direct leakage without the expected normalized action.
  • Unauthorized medical access: Long-Context, Naive RAG, and Policy RAG correctly refuse access to specific patient chart or medical information.Their responses explicitly cite authorization or privacy restrictions.
  • Unauthorized medical access: Mem0 and ReMeM-S disclose that the inquiry concerns the Ortega patient in dermatology, receiving clear failure marks.Both answers identify the patient and specialty rather than refusing the request.
  • Deleted-instruction handling: Policy RAG and ReMeM-I correctly report that deleted instructions are unavailable, while Long-Context, Naive RAG, and A-Mem avoid direct leakage without matching the normalized action.The latter responses are marked !, whereas Policy RAG and ReMeM-I are marked ✓.
  • Deleted-instruction handling: Mem0 clearly leaks the deleted instruction by stating that it asked for House Manager Rosa after 6 PM.This response is marked ✗ in the checkpoint comparison.
Loading 2606.18829v1…