Source-linked AI summary
AIM: A Privacy-Aware Interoperable Memory Framework for Multi-Agent Multi-User LLM Systems
Zachary Johnson, Nigel Boachie Kumankumah, Somya Chatterjee, Tejas Sathyamurthi, Min Chen, Xinyi Alice Li, Xiao Wang, Emily Morgan Gelchie, Jessica Lin, Sadid A. Hasan, Sulaiman Vesal
TL;DR
Session-scoped and single-user memory limits persistent personalization and shared context across users. AIM manages private and shared memories in a multi-agent, multi-user system, and MUMBench evaluates memory operations in that setting. On MUMBench, AIM reports 58.8% strict and 70.5% state-aware operation accuracy, while privacy classification is among its strongest capabilities.
Problem
Existing agent memory systems generally operate at the individual-user level, limiting shared knowledge across users and leaving multi-user privacy-aware memory evaluation unaddressed.
Source: §1 · pp. 1–2Method
AIM combines operation detection, structured memory extraction, privacy classification, scoped deduplication, and visibility-filtered retrieval over a shared memory store.
Source: §2 · p. 3, §3.5 · p. 5Results
AIM achieves 58.8% strict and 70.5% state-aware operation accuracy; visibility classification is its strongest capability, with all five models averaging nearly 90% or greater accuracy.
Source: §5.3 · p. 9Takeaways & Limitations
MUMBench evaluates stateful multi-user memory operations and privacy decisions across four domains, including cases where later interactions depend on earlier memories.
Source: §1 · p. 2, §4 · p. 6Any individual can create or alter public memories with factually incorrect information, and the current public-memory scope is all users.
Source: §6.1 · p. 9
Abstract
from arXiv · showhide
Traditional large language models (LLMs) are scoped to individual user sessions, limiting their knowledge to a single conversation and preventing them from learning user preferences that evolve over time. Existing agentic memory systems address this limitation but generally operate at the individual-user level, restricting the public knowledge that could be shared across users to improve downstream responses. We introduce AIM (Agentic Interoperable Memory), a unified, privacy-aware memory framework that enables multi-agent, multi-user LLM systems to persistently manage private and shared memory. AIM dynamically classifies information as private, scoped to one user and inaccessible to others, or public, accessible to all users. It enforces index-level access controls so that private memories are retrievable only by their owner, protecting sensitive data while allowing beneficial shared knowledge to improve coordination and consistency. We also introduce MUMBench (Multi-User Memory Benchmark), a dataset of multi-user interactions containing private and shareable information across four domains. To our knowledge, MUMBench is the first public dataset designed to evaluate multiple memory operations, including retrieval, creation, update, and deletion, in a multi-user environment. Across three independent runs on MUMBench, AIM achieves 96.0% visibility classification accuracy, 58.8% strict operation accuracy, and 70.5% state-aware operation accuracy.
1 Introduction
AIM addresses the limits of session-bound and single-user memory with a privacy-aware framework for persistent memory across multiple users and agents. The paper also introduces MUMBench to evaluate multi-user memory operations and privacy.
1 Introduction
- AIM is a portable, system-agnostic framework for multi-agent, multi-user systems to manage private and shared memories persistently.Its shared registry is designed to support heterogeneous agents while respecting user-level privacy constraints.Source: §1 · p. 1
- AIM detects memory intent, classifies extracted memories as private or public, and filters retrieval by visibility permissions and user or agent identity.Deduplication and user-initiated deletion help keep the memory store current and retrieval-efficient.Source: §1 · p. 2
- MUMBench evaluates multi-user memory operations across 672 validated interactions, 93 users, and four domains, including stateful lifecycle and privacy decisions.Its interactions are chronologically ordered, so later turns can depend on earlier stored memories.Source: §1 · p. 2
2 Related work
Prior memory systems advance agent memory, modular storage, and sharing, but the cited work leaves gaps in multi-user access control, cross-agent synchronization, and evaluation. AIM and MUMBench target those gaps.
2 Related work
- Earlier agent-memory architectures add reflection, tiered storage, reorganization, and maintenance, but remain fundamentally single-agent and single-user.Generative Agents [Park et al., 2023] introduced observation, reflection, and planning; CoALA [Sumers et al., 2024] formalized memory modules.Source: §2 · p. 2
- Modular and hierarchical memory methods advance memory organization and lifecycle management, while leaving versioning, conflict resolution, and privacy-scoped sharing across users and agents unaddressed.Examples include MIRIX [Wang and Chen, 2025], MemOS [Li et al., 2025], and LEGOMem [Han et al., 2025].Source: §2 · p. 2
- Collaborative Memory [Rezazadeh et al., 2025], INMS [Gao and Zhang, 2024], and MaaS [Li, 2025] address permissions, shared pools, or dynamic access policies; MEXTRA [Wang et al., 2025a] demonstrates private-memory extraction risk.The cited sharing and privacy work does not itself establish a unified multi-user benchmark.Source: §2 · p. 2
- AIM combines automatic visibility classification, deterministic index-level access control, and multi-label operation detection in a modular, system-agnostic service.Unlike pipelines that assume one operation per turn, AIM can recognize simultaneous memory write and retrieval needs.Source: §2 · p. 3
- MUMBench addresses benchmark gaps by evaluating private-versus-shared classification, cross-session synchronization, and memory integrity under concurrent multi-agent updates.MemoryAgentBench [Hu et al., 2025] and LOCOMO [Maharana et al., 2024] are described as single-agent or single-user benchmarks.Source: §2 · p. 3
3 Methods
AIM manages a shared, evolving memory store through write and read paths, with visibility constraints governing access. Its methods cover operation detection, extraction, privacy classification, deduplication, indexing, retrieval, and modularity.
3.1 Problem Formulation
- AIM represents memory records with content, embeddings, ownership, visibility, tags, timestamps, and auxiliary metadata, and applies access constraints to reads and writes.Private memories are visible only to their owner, while public memories are universally accessible; update and delete operations also require ownership.Source: §3.1 · pp. 3–4
3.2 Multi-Label Intent Detection
- AIM uses multi-label intent detection, allowing one message to trigger multiple memory operations such as creating a fact and retrieving relevant memories.The operation detector uses in-context learning and constrained decoding over the operation set.Source: §3.2 · p. 4
3.3 Structured Memory Extraction
- For detected create or update operations, AIM extracts structured memory facts with textual content, keyword tags, and optional temporal or domain metadata.The extractor uses constrained JSON-mode decoding to enforce its output schema.Source: §3.3 · p. 4
3.4 Privacy-Aware Visibility Classification
- AIM classifies each extracted memory as public or private, determining who can access it and the scope of deduplication, retrieval, and update or deletion permissions.The classifier uses prompted LLM classification with public and private signals encoded in the prompt.Source: §3.4 · p. 4
3.5 In-Context Semantic Deduplication
- AIM deduplicates within visibility-restricted comparison sets, classifying each candidate as create, update, or no-op based on semantic overlap and temporal supersession.Private memories are compared only with private memories owned by the same user, avoiding cross-user information leakage through deduplication.Source: §3.5 · p. 5
3.6 Memory Representation and Indexing
- AIM indexes dense embeddings and metadata for approximate nearest-neighbor retrieval and predicate filtering, jointly encoding memory content and tags when tags are present.The tag-enriched representation supports keyword-focused retrieval without a separate sparse index.Source: §3.6 · p. 5
3.7 Cascaded Memory Retrieval
- AIM retrieves memories through index-filtered dense search, a dual content-and-tag threshold, then conservative LLM reranking fused with recency.The retrieval filter excludes other users’ private memories; uncertain reranker matches receive scores of 0.4 or lower.Source: §3.7 · pp. 5–6
3.8 Modularity and Ablation
- AIM defines inference components and the vector index behind abstract interfaces, so components can be swapped or disabled without changing pipeline logic.This configuration-driven modularity supports ablations of components such as deduplication, visibility classification, and tag-enriched embeddings.Source: §3.8 · p. 6
4 MUMBench
MUMBench evaluates multi-user memory systems on private and public information, memory operations, and access control across four domains. The benchmark was generated and manually validated to check interaction quality and privacy handling.
4 MUMBench
- MUMBench evaluates privacy-aware memory classification, access control, and operation selection and execution across four domains.It covers Coding, Customer Support, Education, and Travel Planning.Source: §4 · p. 6
4.1 MUMBench Creation
- GPT-4o generated domain instructions, and researchers manually validated instructions and every sample for chronology, duplicates, operation and payload correctness, privacy, and diversity.Source: §4.1 · p. 7
- 672 interactions span Coding (150), Customer Support (218), Education (160), and Travel Planning (144).Source: §4.1 · p. 7
5 Evaluation
AIM is evaluated by replaying MUMBench interactions through its memory pipeline and scoring operation decisions, content quality, and privacy classification. GPT-5.4-mini leads operation accuracy, while visibility classification is strong across models.
5.1 Experimental Setup
- Five models were evaluated sequentially across four domains through AIM’s pipeline, with the memory store cleared between domains and three trials run per model and interaction.The models served as the primary model across pipeline components; the paper also reports a LOCOMO benchmark in the appendix.Source: §5.1 · p. 7
5.2 Evaluation Methodology
- Strict accuracy requires an exact match between canonicalized gold and predicted operation multisets, after normalizing no-ops and selected multiplicities.Delete multiplicity is preserved because deleting all differs from deleting one.Source: §5.2 · p. 8
- State-aware scoring adjusts expected operations to the live store state, so missed earlier turns do not automatically count against later per-turn decisions.For example, an update to a target never created is relaxed to a create; reads or deletes of missing content become no-ops.Source: §5.2 · p. 8
- Content Quality uses a full-context judge score normalized to 0–1, while Visibility Match checks binary agreement with the expected privacy label.Source: §5.2 · p. 8
5.3 MUMBench Results
- GPT-5.4-mini averaged 58.8% strict, 70.5% state-aware, and 89.5% judge-decision operation accuracy; it also led Content Quality at 91.6%.Source: Table 1 · p. 8, §5.3 · p. 9
- All five models averaged nearly 90% or greater visibility accuracy, with three reaching 100% on Travel.Source: §5.3 · p. 9
- GPT-5.4-mini achieved the highest or near-highest scores on most metrics, suggesting smaller models can remain competitive across memory-management metrics.Source: §5.3 · p. 9
6 Discussion
AIM’s evaluation has limitations in tag extraction and retrieval, and its LOCOMO comparison does not test the multi-user features that distinguish it. Future work targets retrieval, public-memory factuality and scope, and validation of conflict resolution and auditing.
6.1 Limitations
- AIM’s tag overlap is 35.5% on the best model, and retrieval relevance peaks at 45.0%.The LOCOMO evaluation uses a single pass, unlike Mem0’s reported averages over 10 runs, and LOCOMO does not exercise AIM’s multi-user privacy classification or cross-agent synchronization.Source: §6.1 · p. 9
- Because public memories are accessible to all users, any individual can add or edit public memories with factually incorrect information.Source: §6.1 · p. 9
6.2 Future Work
- Future work proposes improving retrieval, adding factual checks at memory creation, scoping public memories to user subsets, and validating conflict resolution and user-initiated auditing.Suggested retrieval directions include tuning the number of returned memories and exploring hybrid keyword-semantic search.Source: §6.2 · p. 9
A.1 Supplementary Metrics
Supplementary metrics report content fidelity and retrieval relevance, alongside procedures for judging decisions and scoring matched memory actions.
A.1 Supplementary Metrics
- When strict and state-aware scores are both zero, a full-context LLM judge assigns exact, defensible, or wrong verdicts to estimate an upper bound on operation accuracy.The judge receives the user input, current mirror contents, and gold and predicted operations.Source: §A.1 · p. 12
- Gold and predicted actions are paired by operation type using greedy best-match selection on sentence-embedding cosine similarity, then scored for content fidelity, visibility match, tag overlap, and retrieval relevance.Retrieval relevance is LLM-judged for read operations; the metrics are computed over matched action pairs.Source: §A.1 · p. 12
- 81.7% Content Fidelity is the average across domains for the highest-performing model, DeepSeek-V3.2.Coding scores highest on average and Customer Support lowest, with the latter attributed likely to the difficulty of extracting complex order preferences.Source: §A.1 · p. 12
- 45.0% Retrieval Relevance is the average score achieved by the best-performing model, GPT-5.4.The passage attributes the lower relevance likely to a fixed pre-filter retrieval count of 5, which can surface low-similarity memories.Source: §A.1 · p. 12
A.4 MUMBench Characteristics
The supplied passages describe MUMBench’s sequential interaction structure, its action and memory records, and dataset checks covering users, operations, visibility, and adversarial queries.
A.4 MUMBench Characteristics
- MUMBench contains 672 sequential interactions, and a later interaction may depend on information written to memory earlier.Source: §A.4 · p. 12
- Each interaction comprises raw user input, a user ID or name, and expected memory actions; each action specifies an operation and associated memory payload fields.Actions include CREATE, READ, UPDATE, DELETE, and NOOP, with fields for newly created, retrieved, updated, or deleted memories as applicable.Source: §A.4 · pp. 12–13
- Reported averages are unweighted four-domain macro-averages computed within each run and aggregated across three independent runs.Content Fidelity and Tag Overlap use matched action pairs; Retrieval Relevance is conditional on matched read operations and can be based on as few as three scored interactions per domain-run.Source: §A.4 · p. 13
- A memory record includes extracted content, a user identifier, keyword tags, and a PRIVATE or PUBLIC visibility label.Source: §A.4 · p. 13
- Dataset checks cover unique users, operation distributions, private versus public memories, and adversarial queries seeking another user’s information.Source: §A.4 · p. 14
A.4.1 Data Attributes
The passage directs readers to Tables 3 and 4 for attributes of the MUMBench dataset.
A.4.1 Data Attributes
- Tables 3 and 4 contain attributes of the MUMBench dataset.Source: §A.4.1 · p. 14
A.5 MUMBench Schema Example {
These appendices document MUMBench’s memory records and visibility cases, AIM’s LOCOMO results and limitations, and guidance for validating and generating synthetic multi-user memory data. The prompts cover domain-specific private and public information, memory operations, and cross-user or adversarial scenarios across coding, customer support, education, and travel.
A.5 MUMBench Schema Example {
- The schema example records a user’s Paris trip and cultural-activity interest as a private memory, alongside user, agent, run, timestamp, and tag fields.The interaction is a query with a create action, and the memory includes creation and update timestamps.Source: §A.5 · pp. 14–15
A.6 MUMBench Ambiguous Example
- A package-delay query illustrates visibility ambiguity: the individual package is personal, but the storm-related delivery delay is public, potentially causing AIM visibility-classification regressions.The example retrieves a public memory stating that a winter storm caused delivery delays in the Northeast US.Source: §A.6 · pp. 15–16
A.7 LOCOMO
- J=0.280 vs. 0.671 for single-hop; AIM underperforms Mem0 across LOCOMO categories, and this single-user benchmark does not test AIM’s multi-user access controls.The authors attribute the lower scores partly to privacy classification and deduplication overhead, possible over-consolidation, and grouping memories by conversation ID; they ran one evaluation, so results may be noisy.Source: §A.7 · p. 16, Table 5 · p. 17
A.8 Data Validation Guidelines
- The validation checklist covers salient, user-grounded create and update operations, explicit reads and deletes, ownership and privacy controls, operation variability, and domain-specific checks.It requires updates only when new information contradicts or builds on an existing memory for that user, and a no-op for attempts to access or delete another user’s memory.Source: §A.8 · pp. 17–18
A.10.1 Base Prompt
- The synthetic-data prompt specifies private and shared memories, memory operations, privacy protection, and a JSON interaction-and-memory schema, while prohibiting unsupported or premature memory operations.It requires memories to come from explicit user-provided information, limits reads to previously written or updated memories, and requires adversarial requests about another user’s private information to be no-ops.Source: §A.10.1 · pp. 21–22
- The base prompt requires realistic, evolving multi-user interactions that cover memory operations, privacy attacks, conflicts, duplicate information, and retrieval across private and shared memories.Adversarial queries must result in no-ops; contradictory information should trigger updates, while repeated information should be a no-op. Private memories cover user-specific information, while public memories represent shared knowledge.Source: §A.10.1 · p. 23
A.10.2 Coding Domain Prompt
- The coding prompt separates developer-specific preferences, projects, and secrets from team-wide versions, conventions, tooling, and practices.It calls for varied language and library scenarios, code examples, debugging with user-provided snippets, adversarial queries about other developers, and realistic shared-version changes over time.Source: §A.10.2 · pp. 25–26
A.10.3 Customer Support Domain Prompt
- The customer-support prompt treats order details, customer issues, preferences, and history as private, while policies, product information, and common issues can be shared.Scenarios include changing order statuses and issue outcomes, external causes affecting multiple customers, and adversarial queries about other customers’ data.Source: §A.10.3 · p. 27
A.10.4 Education Domain Prompt
- The education prompt supports multiple actions per input, retrieves all relevant active memories, and preserves distinct study focuses unless a student explicitly requests deletion.It distinguishes student-specific learning and wellbeing information from public educational knowledge; a new focus creates a memory rather than overwriting an earlier topic.Source: §A.10.4 · p. 28
- Education deletion requests must target the requested memories, while knowledge-vault scenarios test sharing and queries about other students’ private information.Broad requests can delete multiple relevant memories; examples probe others’ study topics, difficulties, and shared knowledge-vault retrieval.Source: §A.10.4 · pp. 28–29
- Education scenarios span academic topics, formulas, programming, varied learning needs, wellbeing, and study-topic changes.They also require realistic formulas, code, equations, and worked examples.Source: §A.10.4 · p. 29
A.10.5 Travel Planning Prompt
- Travel prompts distinguish private trip details from shared advice and cover planning, budgets, bookings, requirements, itineraries, updates, and local information.Private examples include destinations and dates, budgets, preferences, booking references, companions, and visa or passport status; adversarial queries probe other travelers’ details.Source: §A.10.5 · p. 30