PaperlayerView exact revision on arXiv

Source-linked AI summary

Collaborative Memory: Multi-User Memory Sharing in LLM Agents with Dynamic Access Control

Alireza Rezazadeh, Zichao Li, Ange Lou, Yuying Zhao, Wei Wei, Yujia Bao

arXiv:2505.18279v1cs.MAcs.AIcs.CLcs.LG

TL;DR

Existing LLM memory approaches largely assume a single-user setting, leaving cross-user knowledge transfer under asymmetric, changing permissions unaddressed. Collaborative Memory combines dynamic access graphs, private and shared memory, and fine-grained read/write policies. Across three progressively complex scenarios, it reports lower resource utilization than single-user baselines while maintaining strict privacy compliance.

  • Problem

    Most existing memory designs assume a single-user context, leaving the benefits and challenges of cross-user transfer under asymmetric, changing permissions unaddressed.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    Collaborative Memory combines dynamic permission graphs, private and shared memory with provenance, and fine-grained read/write policies for multi-user, multi-agent knowledge sharing.

    Source: §1 · pp. 2–3, §3.2 · p. 5

  • Results

    Across three progressively complex scenarios, the framework reports lower resource utilization than single-user baselines while maintaining strict privacy compliance.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The framework supports cross-user knowledge sharing while respecting evolving permissions and maintaining privacy compliance in the evaluated scenarios.

    Source: §7 · p. 9

    The evaluation uses simulated multi-user environments and moderate numbers of users and agents, leaving real-world complexity and large-scale enterprise settings unexplored.

    Source: §6 · pp. 8–9

Abstract

from arXiv · show

Complex tasks are increasingly delegated to ensembles of specialized LLM-based agents that reason, communicate, and coordinate actions-both among themselves and through interactions with external tools, APIs, and databases. While persistent memory has been shown to enhance single-agent performance, most approaches assume a monolithic, single-user context-overlooking the benefits and challenges of knowledge transfer across users under dynamic, asymmetric permissions. We introduce Collaborative Memory, a framework for multi-user, multi-agent environments with asymmetric, time-evolving access controls encoded as bipartite graphs linking users, agents, and resources. Our system maintains two memory tiers: (1) private memory-private fragments visible only to their originating user; and (2) shared memory-selectively shared fragments. Each fragment carries immutable provenance attributes (contributing agents, accessed resources, and timestamps) to support retrospective permission checks. Granular read policies enforce current user-agent-resource constraints and project existing memory fragments into filtered transformed views. Write policies determine fragment retention and sharing, applying context-aware transformations to update the memory. Both policies may be designed conditioned on system, agent, and user-level information. Our framework enables safe, efficient, and interpretable cross-user knowledge sharing, with provable adherence to asymmetric, time-varying policies and full auditability of memory operations.

1 Introduction

Persistent memory supports long-horizon reasoning in LLM agents, but existing approaches largely assume a single user and globally accessible memory. The paper introduces a framework for multi-user, multi-agent memory under asymmetric and changing permissions.

1 Introduction

  • Persistent memory approaches support LLM agents’ long-horizon reasoning, while multi-agent systems coordinate specialized agents and external resources.Distributed Cognition frames cognitive processes as distributed across people, artifacts, and their interactions.Source: §1 · pp. 1–2
  • Single-user, globally accessible memory assumptions leave multi-user systems with information asymmetry and permissions that change over time underexplored.Different users may access different agents and resources, so memory sharing must respect who may access what and when.Source: §1 · p. 2
  • Collaborative Memory combines dynamic user-agent and agent-resource permission graphs with private and shared memory tiers and fine-grained read and write policies.The policies can be specified at system, agent, and user levels and adapt over time.Source: §1 · pp. 2–3
  • The paper presents its framework as the first formulation of memory sharing explicitly accounting for fine-grained access asymmetries in multi-user, multi-agent systems.It lays foundational concepts and formalizations for future research into efficient, secure, adaptive collaborative multi-agent systems.Source: §1 · p. 3

2 Related Work

Prior work develops persistent memory for individual agents, multi-agent collaboration, and access-control models, but does not address dynamic, asymmetric permissions for users collaborating with multiple LLM agents. Collaborative Memory connects these threads through access-control graphs and policy-conditioned memory transformations.

2 Related Work

  • RBAC and ABAC provide permission-management abstractions, while this framework models time-varying user-agent and agent-resource graphs and provenance-aware memory fragments.The paper states that no existing LLM memory system combines these policies with provenance-aware fragment storage.Source: §2 · p. 3
  • Prior work on persistent memory for individual agents and multi-agent collaboration does not accommodate dynamic, asymmetric permissions when multiple users interact with multiple LLM agents.Memory Sharing supports a common memory pool but overlooks user-level privacy and access constraints.Source: §2 · p. 3
  • Collaborative Memory bridges structured individual-agent memory and multi-agent collaboration with formal access-control graphs and policy-conditioned read/write transformations.The stated aim is safe and auditable cross-user knowledge transfer.Source: §2 · pp. 3–4

3 Collaborative Memory

Collaborative Memory manages cross-user knowledge sharing with dynamic access constraints. Its design combines permission graphs, provenance-bearing private and shared memory, and read and write policies that tailor access and storage.

3 Collaborative Memory

  • The framework uses user queries and agent responses with write policies to update private and shared memories, while later agents retrieve only permitted information.The framework is presented as enabling cross-user knowledge transfer without violating user or agent access constraints.Source: §3 · p. 4

3.1 Setup and Dynamic Access Graphs

  • At each time step, user-agent and agent-resource bipartite graphs determine which agents a user may invoke and which resources an agent may access.The graphs evolve to represent new users, revocations, and changes in resource permissions.Source: §3.1 · p. 4

3.2 Memory Tiers and Provenance

  • Each memory fragment carries immutable creation-time, contributing-user, contributing-agent, and accessed-resource provenance, and fragments are divided into private and cross-user-shareable stores.The system uses agent- and resource-level constraints while allowing cross-user and cross-agent sharing.Source: §3.2 · p. 5

3.3 Read from the Collaborative Memory

  • A query is routed to suitable accessible agents, whose read policies filter relevant memory before responses are generated and optionally aggregated.Read policies may limit retrieved fragments or filter them by keywords.Source: §3.3 · p. 5

3.4 Write to the Collaborative Memory

  • After an agent produces an output, separate private and shared write policies determine what is stored and what confidentiality transformations are applied.Policies may anonymize entities, redact sensitive information, or block content from storage.Source: §3.4 · pp. 5–6

4 Implementation Details

The implementation coordinates specialized agents through a query-driven loop and stores retrieved task knowledge as provenance-tagged memory fragments. Retrieval and read/write policies control which fragments are available, while performance is tracked through accuracy and usage metrics.

4 Implementation Details

  • The coordinator selects accessible agents for subqueries, can stop the loop early, and an aggregator synthesizes the agents’ responses into a final answer.Resources are exposed as callable tools with JSON schemas.Source: §4 · p. 6
  • The coordinator receives the query, accessible-agent specializations, and conversation history, then emits agent/subquery instructions or a stop signal.Source: §4 · p. 6
  • After a subtask, traces become candidate key–value fragments annotated with contributing agents, accessed resources, and creation time.Source: §4 · p. 6
  • Retrieval selects top-k user-specific and cross-user fragments that satisfy provenance constraints, ranking them by cosine similarity to the subquery embedding.Source: §4 · p. 6
  • Read policies can return admissible fragments verbatim, while write policies can store fragments unchanged or transform them through prompts; experiments use simple reads and transformation writes.Source: §4 · p. 6
  • Accuracy measures normalized correctness, agent utilization counts distinct agents per query, and resource utilization counts knowledge-base or API calls per query as an efficiency proxy.Source: §4 · p. 6

5 Experimental Evaluation

The evaluation tests fully shared, asymmetric, and dynamically changing access scenarios, tracking accuracy, resource use, and privacy compliance. Collaborative memory reduces resource use in the reported scenarios, while response quality changes with resource availability.

5 Experimental Evaluation

  • The evaluation covers three scenarios and tracks accuracy, resource consumption, and privacy compliance; direct latency assessment is challenging because agent latency fluctuates substantially.The cited latency range is 5–30 minutes.Source: §5 · p. 6

5.1 Scenario 1: Fully Collaborative Memory

  • Up to 61% fewer resource calls at 50% overlap and 59% at 75% versus isolated memory; accuracy remained above 0.90 on average.Source: Figure 3 · p. 7, §5.1 · p. 7
  • In the fully collaborative scenario, five users queried six domain-specialist agents over a partitioned MultiHop-RAG corpus, with query overlap varied across users.The dataset contains 2,556 multi-hop questions from 609 English news articles spanning six domains.Source: §5.1 · p. 7

5.2 Scenario 2: Asymmetric Collaborative Memory

  • The asymmetric scenario used 200 business-project queries and four user roles with distinct expertise; only the Strategy Director could consult all four specialized agents.Source: §5.2 · p. 7
  • Asymmetric collaboration reduced overall resource calls versus isolation as insights flowed between users with matching privileges, avoiding repeated tool or knowledge-base queries.By the Strategy Director’s consolidation of recommendations, partial sharing had already eliminated a significant portion of redundant work.Source: Figure 4 · p. 8, §5.2 · p. 8

5.3 Scenario 3: Dynamically Evolving Collaborative Memory

  • The dynamic-access simulation used five users and resources, adding permission edges and then revoking them across time steps while reusing the same 100 queries.Edges were added through Bernoulli trials, with grants and revocations representing changing access.Source: §5.3 · p. 8
  • With changing permissions, accuracy and agent use rose as access was granted and declined when privileges were revoked; queried resources tended to decrease as memory fragments were reused.The access matrix showed users accessing only agents and resources granted by the graph.Source: §5.3 · p. 8

6 Limitations

The limitations concern the realism and scale of the evaluation, possible LLM errors despite enforcement, and simplified resource-utilization measurement.

6 Limitations

  • The evaluation relies on existing benchmarks or synthetic queries in simulated multi-user environments, which may not fully reflect real-world complexity.The authors identify more realistic multi-user collaboration benchmarks as a worthwhile direction.Source: §6 · pp. 8–9
  • Controlled settings with moderate numbers of users and agents leave large-scale enterprise settings with frequent concurrency and rapidly evolving roles unexplored.Source: §6 · p. 9
  • LLMs’ probabilistic behavior can lead to occasional hallucinations or policy breaches despite the framework’s enforcement mechanisms.Source: §6 · p. 9
  • Resource utilization is measured by resource-call counts because production API latencies are unpredictable.Source: §6 · p. 9

7 Conclusions

The conclusion presents Collaborative Memory as a framework for multi-user, multi-agent memory sharing under asymmetric, changing access constraints. Across three scenarios, it reports lower resource utilization than single-user baselines while maintaining strict privacy compliance.

7 Conclusions

  • Across three progressively complex scenarios, the framework reports significantly lower resource utilization than single-user baselines while maintaining strict privacy compliance.Source: §7 · p. 9
  • Collaborative Memory combines dynamic bipartite access graphs, a two-tier memory system, and fine-grained read/write policies to address information asymmetry and changing access constraints.Source: §7 · p. 9
  • The conclusion characterizes the design as supporting secure, context-aware knowledge sharing that respects evolving permissions.Source: §7 · p. 9

Disclaimer

The disclaimer disclaims professional-advice substitution and endorsement of third-party marks. The appendix outline lists discussions of impact, pipeline and policies, and three collaborative-memory scenarios with their supporting materials.

Disclaimer

  • The disclaimer says the document is for general information and is not a substitute for consulting professional advisors.Source: Disclaimer · p. 10
  • It states that third-party marks remain their owners’ property and disclaims sponsorship, endorsement, or approval by those owners.Source: Disclaimer · p. 10

Appendix Arrangement

  • The appendix outline covers the broad-impact discussion, query-to-response pipeline, read/write policies, and fully collaborative, asymmetric, and dynamically evolving scenarios.It also lists scenario materials such as dataset details, experimental setups, access graphs, interactions, and performance data.Source: Appendix Arrangement · p. 12

A Discussion of the Broad Impact of Collaborative Memory

The discussion presents Collaborative Memory as a modular framework for permission-aware knowledge sharing, combining dynamic access graphs, private and shared memory, and configurable policies. Its appendix pipeline selects authorized agents, uses their responses to update memory, and aggregates outputs; read and write policies govern memory access and transformation.

A Discussion of the Broad Impact of Collaborative Memory

B.1 Pipeline

  • The pipeline selects eligible agents using the query and access graph, gathers their responses, updates collaborative memory, and aggregates intermediate outputs into a final response.The current implementation updates memory using intermediate responses; final responses can also be used.Source: §B.1 · p. 13
  • In the current implementation, the Coordinator and Aggregator use the GPT-4o API.Source: Figure 7 · p. 14

B.2 Read/Write Policy

  • The read policy combines accessible private and shared fragments, while system-level LLM prompts transform memory for private and shared writing.The described unified policy can be extended to policies customized for individual users or agents.Source: §B.2 · p. 14

C.1 Dataset Details

The experiments cover multi-user memory sharing across datasets, access configurations, and query types, including a role-based asymmetric scenario. They report reduced resource use with shared memory and describe how stored fragments are reused across queries and users.

C.1 Dataset Details

  • MULTIHOP-RAG contains 609 news articles averaging approximately 2,046 tokens and 2,556 multi-hop queries across four query types.The query types are inference, comparison, temporal reasoning, and null-query instances.Source: §C.1 · p. 15, Table 2 · p. 15

C.2 Experimental Details

  • Four multi-user datasets vary global query overlap from 0% to 75%, comparing shared and isolated memory with user-specific and shared tier sizes of 10.Each configuration uses k_user = 10 and k_cross = 10.Source: §C.2 · p. 15

C.2.1 Access Graph Configuration

  • Five users can invoke six specialist agents, while each agent is limited to one domain-aligned resource; these access graphs remain fixed during experiments.The fixed graphs isolate the effects of memory sharing.Source: §C.2.1 · p. 15, §C.2.2 · p. 15

C.2.2 Agent/Knowledge Base Configuration

  • The six specialist agents use dedicated knowledge bases containing between 10 health documents and 211 sports documents, with precomputed embeddings.Table 3 lists agent descriptions and system prompts.Source: §C.2.2 · p. 15, Table 3 · pp. 16–17

C.3 Examples

  • Later queries can reuse stored fragments without new knowledge-base searches; one example shares a cryptocurrency-trial fragment from user_2 with user_3.The examples include both user-specific and cross-user memory entries.Source: §C.3 · pp. 17–18, Table 4 · p. 17

C.4 Performance Breakdown by Query Type

  • Shared memory reduces agent calls and external retrievals across query types, with the largest resource-efficiency benefit for temporal queries while maintaining comparable accuracy.Inference accuracy is near ceiling in both conditions; comparison tasks show a small accuracy gain under sharing.Source: §C.4 · p. 18, Figure 8 · p. 19

D Scenario 2: Asymmetric Collaborative Memory

D.1 Dataset Details

  • The GPT-4o-synthesized business-scenario dataset contains 200 queries, evenly split between 100 easy and 100 hard cases.The generation prompt specifies a project query and four user roles: market, finance, logistics, and decision maker.Source: §D.1 · p. 20

D.2 Experimental Details

This section describes experiments in a simulated multi-user, multi-agent setup, using domain-specialized agents, resources, and coordination to illustrate tasks across market, finance, logistics, and strategy domains. It examines collaboration under asymmetric, evolving access and memory reuse through example interactions and projections, rather than comparative system-evaluation results.

D.2 Experimental Details

  • Experiments compare memory-sharing and non-sharing settings, with user and shared retrieval tiers of 20 each and four users assigned varying access to agents and resources.The access structure is hierarchical: agents also have controlled access to task-relevant resources.Source: §D.2 · p. 21
  • The pool has four domain agents—market_agent, finance_agent, logistics_agent, and decision_agent—using GPT-4o-simulated resources for market knowledge, financial forecasts, logistics comparison, and strategic reasoning.The resource names are market_kb, finance_forecaster, logistics_comparator, and strategic_computation.Source: §D.2 · p. 21
  • Illustrative calls cover European skincare demand, logistics and supply-chain assessment, cross-agent synthesis, and a 5% price-increase margin calculation assuming constant costs and low demand elasticity.The decision agent is prompted to synthesize market, finance, and logistics insights; the strategic computation example asks for a new profit margin.Source: §D.2 · pp. 21, 22, 24
  • The projection estimates first-year revenue of EUR 5,000,000 from approximately 100,000 units sold across all markets at an average retail price of EUR 50 per unit.Source: §D.2 · pp. 23–24

D.2.1 Access Graph Configuration

  • The access graph encodes asymmetric user-to-agent and agent-to-resource permissions, and only the strategy director has comprehensive visibility and access.Low-level users handle role-specific subtasks, while the strategy director follows those roles to synthesize the final response.Source: §D.2 · p. 25, §D.2.1 · p. 25, Figure 10 · p. 26

E.2.2 Agent/Knowledge Base Configuration

  • Domain-specialized agents use expertise-tailored prompts and relevant resources, while a coordinator assigns queries and an aggregator combines outputs.The configurations cover domain agents, coordinator and aggregator roles, and resources such as knowledge bases, forecasting tools, and computation modules.Source: §D.2.2 · pp. 26–27, Table 7 · p. 27, Table 8 · p. 27

D.3.1 Read/Write example

  • A collaboration example shows agents reading and writing both private user-specific and public cross-user memory to support continued reasoning and synthesis.The examples demonstrate memory reuse across users and roles.Source: §D.3.1 · p. 27, Table 10 · p. 28

D.3.2 Without Subtask Decomposition

  • Role-aligned reformulation enables a detailed response, while the original high-level query fails when limited agent and resource access misaligns with the task.The example illustrates how coordinated collaboration among role-specialized users, agents, and resources addresses complex tasks.Source: §D.3.2 · p. 28, Table 11 · p. 29

E.1 Dataset Details

  • The SciQAG experiment selects five categories from 24 Web of Science categories to evaluate behavior under dynamic access without using the full dataset.Each instance pairs scientific-paper passages with question–answer pairs; Table 12 reports document counts, average lengths, and query-type distributions.Source: §E.1 · p. 30, Table 12 · p. 30

E.2 Experimental Details

  • The dynamic experiment uses 100 queries from five users, five category-specialist agents, and five corresponding resources, with shared memory enabled.It reuses the same queries across graph configurations and sets k_user = 10, k_cross = 10, and the similarity threshold to 0.1.Source: §E.2 · p. 30

E.2.1 Access Graph Configuration

  • Access evolves from granting 5 edges per stage to revoking 5 per stage across t0–t8, with a one-to-one agent–resource mapping and p = 0.2.The graph grows to 25 edges at t4, then shrinks to 5 at t8; each stage's graph becomes the next stage's base.Source: §E.2.1 · p. 30, Table 13 · p. 32, Table 14 · p. 33

E.3 Examples

  • The memory examples include private reuse by user_1 from Query #18 to Query #40 and shared reuse of user_5’s stored information by user_2.Each case identifies the memory entry, its triggering query, and the later query that retrieves it.Source: §E.3 · p. 36

E.4 Raw Performance Data and Complete Access Matrix Under Different Access Graph Configurations

The section presents raw performance data across eight time blocks and a complete access matrix under changing privileges. The matrix shows agent and resource use only where access is granted.

E.4 Raw Performance Data and Complete Access Matrix Under Different Access Graph Configurations

  • The section also reports raw performance across eight time blocks, private and shared memory examples, and performance and memory statistics across access-graph configurations.Tables 18 and 19 cover memory write–read examples and cross-configuration performance and memory statistics, respectively.Source: §E.4 · p. 36, Table 18 · p. 37, Table 19 · p. 37
  • Figure 11 shows agent and resource use only in explicitly granted access cells, even as privileges change.Yellow rectangles indicate granted access, and their values represent usage counts.Source: §E.4 · p. 36, Figure 11 · p. 38
Loading 2505.18279v1…