PaperlayerView exact revision on arXiv

Source-linked AI summary

LatentMem: Customizing Latent Memory for Multi-Agent Systems

Muxin Fu, Xiangyuan Xue, Yafu Li, Zefeng He, Siyuan Huang, Xiaoye Qu, Yu Cheng, Yang Yang

arXiv:2602.03036v2cs.CLcs.LGcs.MA

TL;DR

Existing multi-agent memory systems face role-insensitive homogenization and information overload. LatentMem addresses these issues with role-aware latent memory customization, and experiments show improved performance, efficiency, and generalization across benchmarks and MAS frameworks.

  • Problem

    Existing multi-agent memory systems use one-size-fits-all memories and increasingly large stores of fine-grained entries, which can undermine role adherence and obscure critical decision signals.

  • Method

    LatentMem combines a lightweight experience bank, an agent-profile-conditioned memory composer, and Latent Memory Policy Optimization to produce compact, role-aware latent memories.

  • Results

    LatentMem consistently outperforms existing memory designs across six benchmarks and four MAS frameworks, improving performance by up to 16.20% and 18.45% in knowledge QA and code generation, respectively.

  • Takeaways & Limitations

    Role-aware, fixed-length latent memories and differentiable optimization support efficient memory customization and generalization across task domains and MAS frameworks.

  • Takeaways & Limitations

    Real-world deployment requires appropriate oversight, safety assessments, and compliance because more powerful memory capabilities may increase misuse potential.

Abstract

from arXiv · show

Large language model (LLM)-powered multi-agent systems (MAS) demonstrate remarkable collective intelligence, wherein multi-agent memory serves as a pivotal mechanism for continual adaptation. However, existing multi-agent memory designs remain constrained by two fundamental bottlenecks: (i) memory homogenization arising from the absence of role-aware customization, and (ii) information overload induced by excessively fine-grained memory entries. To address these limitations, we propose LatentMem, a learnable multi-agent memory framework designed to customize agent-specific memories in a token-efficient manner. Specifically, LatentMem comprises an experience bank that stores raw interaction trajectories in a lightweight form, and a memory composer that synthesizes compact latent memories conditioned on retrieved experience and agent-specific contexts. Further, we introduce Latent Memory Policy Optimization (LMPO), which propagates task-level optimization signals through latent memories to the composer, encouraging it to produce compact and high-utility representations. Extensive experiments across diverse benchmarks and mainstream MAS frameworks show that LatentMem achieves a performance gain of up to $19.36$% over vanilla settings and consistently outperforms existing memory architectures, without requiring any modifications to the underlying frameworks.

1. Introduction

LLM-powered multi-agent systems use memory to retain and reuse experience for coordination and adaptation, but existing designs face role-insensitive homogenization and information overload. LatentMem addresses these constraints with role-aware latent memories, a lightweight experience bank, a memory composer, and LMPO, while experiments report gains across benchmarks and frameworks.

  • Multi-agent memory helps agents accumulate, retain, and reuse interaction experience for more coherent coordination and continual adaptation.
  • Existing memory systems suffer from memory homogenization because one-size-fits-all designs ignore agents’ functional heterogeneity.This can undermine role adherence, amplify correlated errors, weaken robustness, and hinder long-term adaptation.
  • Multi-granularity memory designs can create information overload by storing large volumes of entries alongside long MAS interaction contexts.The resulting burden can overwhelm agents and obscure critical decision signals.
  • LatentMem combines a lightweight experience bank with a memory composer that uses agent profiles to distill raw trajectories into compact, role-aware latent memories.LMPO trains the composer using relative-reward advantages, token-level objectives, and gradient backpropagation through differentiable latent memories.
  • LatentMem uses fixed-length latent representations and role profiles to mitigate information overload and memory homogenization while avoiding handcrafted memory architectures.
  • 16.20% and 18.45% improvements over state-of-the-art MAS are reported for knowledge QA and code generation, respectively, alongside 50% fewer tokens and broader generalization gains.Reported generalization gains include 7.10% on PDDL and 7.90% on unseen CAMEL compared with the vanilla setting.

2. Related Works

Related work spans adaptive MAS organization and multi-agent memory, but existing approaches can incur substantial computational or token overhead. LatentMem’s framework overview presents a lightweight memory pipeline for retrieving, composing, injecting, and continually storing agent-specific memories.

  • Modern MAS methods dynamically reconfigure organizational structures to improve adaptability across complex tasks while reducing computational costs.
  • These adaptive MAS methods typically require extensive design-space searches, creating considerable computational and token overhead.Developing a lightweight mechanism for flexible MAS adaptation therefore remains an open research problem.
  • LatentMem retrieves relevant trajectories, compresses them with role profiles through an LMPO-trained composer, injects latent memories without changing agent architectures, and stores new trajectories.
  • Earlier MAS memory designs commonly used simple within-trial mechanisms, including shared pools that store raw trajectories.Memory supports experience accumulation, coherent coordination, and continual adaptation.

3. Preliminary

The preliminary formulation models each MAS agent by a role profile and policy, with execution governed by an agent topology and memory retrieval. The paper frames memory design as maximizing expected task performance and motivates a learnable module that generates compact, role-aware latent representations.

  • A multi-agent system contains agents with role profiles and policies, receives prompts and retrieved memory, and produces responses under an execution graph.The graph may be static or centrally regulated dynamically, depending on the architecture.
  • The objective is to find a memory module that maximizes the expected performance of the multi-agent system over dataset queries.
  • For each query, the system generates a reasoning trajectory whose final answer is evaluated by a reward function for correctness.
  • The formulation supports both handcrafted symbolic and learnable parameterized memory architectures.LatentMem adopts a learnable module that generates compact, role-aware latent representations for dynamic integration into agent reasoning.

4. Methodology

LatentMem combines a lightweight trajectory experience bank with a role-conditioned memory composer that produces compact latent memories for multi-agent reasoning. LMPO optimizes the composer through task-level feedback propagated across a differentiable memory interface.

  • Overall Framework: LatentMem retrieves relevant historical trajectories, combines them with each active agent’s role profile, and distills them into compact latent memories.The experience bank stores raw trajectories, while the composer produces fixed-length, agent-aware representations for reasoning.
  • Memory Composer: The composer injects role-aware latent memories into token embeddings through a model-level wrapper, requiring no modifications to the underlying system architecture.The resulting extended input combines prompt hidden states with the fixed-length latent memory.
  • Experience Bank: The experience bank stores raw MAS trajectories, retrieves K relevant trajectories by similarity, and appends completed trajectories for online reuse.Retrieval maps queries and trajectories into a latent embedding space and compares them with cosine similarity.
  • Latent Memory Policy Optimization (LMPO): LMPO uses task-level feedback through differentiable latent memories to optimize the memory composer while keeping agent backbones frozen.The composer’s parameters influence agent outputs through the latent memory interface, allowing end-to-end optimization.
  • Latent Memory Policy Optimization (LMPO): Token-level surrogate optimization addresses the reduced contribution of tokens in longer multi-agent interactions under trajectory-level objectives.The token-level importance sampling ratio measures how updated memories modulate an active agent’s policy at each reasoning step and token.

5. Experiments

Across six benchmarks and four MAS frameworks, LatentMem improves performance, generalization, and resource efficiency while producing role-specific memories and benefiting from both its composer and experience bank.

  • Experimental Setup: The evaluation spans six benchmarks across knowledge-intensive QA, coding, reasoning QA, and symbolic planning, using AutoGen, MacNet, CAMEL, and DyLAN with two LLM backbones.Baselines include memory-free methods, single-agent memory systems, and multi-agent memory architectures.
  • Main Results: LatentMem outperforms representative memory baselines by average gains of 7.86% against single-agent methods and 6.66% against multi-agent methods, including a 16.20% improvement for AutoGen on TriviaQA.Using Llama-3.1-8B-Instruct, MacNet performance on KodCode rises from 48.50% to 65.50%.
  • Main Results: 7.10% improvement on AutoGen PDDL and 7.05% improvement on CAMEL KodCode show stronger generalization than baselines across out-of-domain benchmarks and unseen MAS frameworks.MetaGPT and Voyager drop by up to 4.44% and 2.77%, respectively, on PDDL.
  • Cost Analysis: LatentMem combines larger performance gains with lower resource costs, including 11.68% over no-memory on DyLAN TriviaQA and 8.40% on AutoGen KodCode.It cuts inference time by 2.16× relative to OAgents on the cited setting and uses 0.01M fewer tokens than no-memory for AutoGen KodCode.
  • Sensitivity & Ablation Study: Role-specific latent memories remain separated across in-domain, out-of-domain, seen, and unseen settings, while cumulative accuracy improves as more experiences are collected.LatentMem surpasses complex multi-granularity baselines after early higher-variance performance.
  • Sensitivity & Ablation Study: Performance generally improves with latent memory length but shows diminishing returns, with L′ = 8 selected; removing role guidance or real-time experience updates causes setting-dependent degradation.The largest reported drops are 6.45% on MacNet KodCode without role guidance and 7.63% on PDDL without experience updates.
  • Sensitivity & Ablation Study: LatentMem’s role-aware latent memory helps prevent step repetition and blind trajectory following, enabling role compliance, coordination, and short-term self-correction.The case study contrasts vanilla MacNet and MacNet with OAgents against LatentMem.

6. Conclusion

LatentMem is a latent memory framework that customizes multi-agent memories by role while improving token efficiency. Its experience bank, memory composer, and LMPO training support performance, generalization, and efficiency across benchmarks and MAS frameworks.

  • LatentMem enables role-aware and token-efficient memory customization in multi-agent systems.Each agent receives latent memories distilled from raw trajectories, reinforcing role compliance and coordination.
  • LatentMem combines a lightweight experience bank with a learnable memory composer to distill raw trajectories into agent-specific latent memories.LMPO uses task-level feedback to encourage transferable, high-utility representations.
  • LatentMem achieves substantial performance gains, robust generalization, and high efficiency across six benchmarks and four MAS frameworks.The framework is reported to mitigate memory homogenization and information overload without modifying the underlying MAS frameworks.

Impact Statement

The paper positions its contribution within controlled, offline research using public benchmarks and established MAS frameworks. It notes potential benefits for applications but stresses safeguards for real-world deployment.

  • The study uses publicly accessible benchmarks and datasets in controlled, offline research environments without private or sensitive user information.The authors report no major ethical risks from the study itself.
  • The proposed methods target robustness and reliability improvements for LLM-based multi-agent systems, with potential applications in assistive robotics, information organization, and long-term decision-making.The authors frame these as potential benefits rather than deployment outcomes.
  • More powerful memory capabilities may increase misuse potential if deployed without proper safeguards.The paper calls for oversight, safety assessments, and compliance with legal and ethical standards in real-world deployment.
  • The evaluation uses datasets including TriviaQA, KodCode, StrategyQA, PopQA, BigCodeBench, and PDDL for question answering, coding, reasoning, and planning tasks.The datasets span in-domain and out-of-domain evaluation settings.
  • The experiments cover AutoGen, MacNet, CAMEL, and DyLAN as adopted multi-agent system frameworks.The supplied passages describe their orchestration, decentralization, role-playing, and debate-style designs.

B.4.1. Evaluation of Generalization

Generalization is evaluated across unseen domains and unseen multi-agent systems, using training data collected from in-domain tasks and in-distribution MAS configurations. The section also records implementation settings and parameter details.

  • Evaluation of Generalization: Generalization is evaluated along two dimensions: domains unseen during training and multi-agent systems unseen during training.TriviaQA, KodCode, StrategyQA, and PopQA are in-domain, while BigCodeBench and PDDL are out-of-domain; AutoGen and MacNet are encountered MAS.
  • Evaluation of Generalization: The training corpus contains 40,580 trajectories collected from AutoGen and MacNet on four in-domain datasets.Training data comes from the training splits of TriviaQA, KodCode, StrategyQA, and PopQA.
  • Evaluation of Generalization: LatentMem uses a pretrained LLM as its memory composer, with training based on raw MAS trajectories collected on AutoGen and MacNet.The supplied implementation passage names Qwen3-4B-Instruct-2507 and Llama-3.1-8B-Instruct as examples.
  • Evaluation of Generalization: Detailed LatentMem hyperparameter configurations are reported in Table 3.The supplied passages identify the table as containing detailed parameter settings.

C.1. Cost Analysis

LatentMem’s latent memory design is reported to reduce inference resource costs while producing substantial performance improvements across benchmarks and MAS frameworks.

  • LatentMem achieves the fastest inference and lowest token usage among memory baselines while delivering the most substantial improvements.Figure 8 compares token and time costs across benchmarks, MAS frameworks, and memory architectures.

C.2. Role-aware Memory Analysis

The analysis represents each latent memory sequence as a compact mean embedding, then applies dimensionality reduction to visualize its organization. Figure 8 compares performance with resource costs, while Figures 9 and 10 examine latent-memory structure and retrieval behavior.

  • Efficiency analysis: Figure 8 shows performance-resource trade-offs using time and token cost, with circle area indicating relative resource consumption.The top row plots performance versus time, and the bottom row plots performance versus token cost; “A + B” denotes framework A evaluated on dataset B.
  • Representation and visualization: Each latent memory sequence is converted into a fixed-size vector by computing the mean of its token embeddings.The sequences contain L′ token embeddings of dimensionality D, and the resulting vectors support downstream analysis.
  • Representation and visualization: t-SNE maps the resulting high-dimensional memory vectors into two dimensions for scatter-plot visualization.The two-dimensional embeddings are used to inspect the geometric organization of latent memories.
  • Representation and visualization: Figure 9 visualizes latent memories generated across different datasets and multi-agent-system frameworks.The figure is used to examine the organization of latent memories across experimental settings.
  • Retrieval sensitivity: Figure 10 compares LatentMem and G-Memory as the number of retrieved trajectories varies on KodCode and TriviaQA with AutoGen.LatentMem benefits from increasing K, whereas G-Memory declines when K > 3.

C.3. Sensitivity Analysis

The sensitivity analysis studies how the number of retrieved raw trajectories affects memory performance. LatentMem continues improving as K increases, whereas G-Memory declines beyond K = 3, indicating different responses to additional trajectories.

  • Sensitivity Analysis: LatentMem continues improving as the number of retrieved raw trajectories increases.The analysis attributes this behavior to fixed-length, role-aware latent memory that can distill useful information from redundant trajectories.
  • Sensitivity Analysis: 72.95% at K = 3 falls to 70.25% at K = 5 for G-Memory on KodCode.G-Memory initially benefits from increasing K, but its performance starts declining when K > 3.
  • Broader evaluation: Table 4 compares diverse memory frameworks across six benchmarks and four multi-agent-system frameworks.The benchmarks include in-domain and out-of-domain tasks, while the frameworks include in-distribution and unseen settings.

E. Example

The examples illustrate LatentMem-enhanced agents completing Knowledge QA and PDDL tasks through coordinated reasoning and valid actions. In the PDDL trajectory, the agents move and manipulate objects until the task is completed.

  • Example setup: The example section presents LatentMem-enhanced AutoGen trajectories for TriviaQA and PDDL tasks.These examples are intended to demonstrate the effectiveness of LatentMem on Knowledge QA and Game tasks.
  • Successful trajectory: A successful trajectory moves the robot between rooms, picks up objects with available grippers, and drops them at the target location.The trajectory includes moving from roomb to rooma, picking up ball3, returning to roomb, and dropping it.
  • Successful trajectory: The demonstrated PDDL trajectory ends with the task completed after ball3 is dropped in roomb.The final state records ball3 at roomb, both grippers free, and the task completed.
  • PDDL action rules: The PDDL domain permits think, move, pick, and drop actions with explicit preconditions and state-update effects.Move requires the robot to be in a room; pick requires co-location and a free gripper; drop requires carrying the object and being in a room.
Loading 2602.03036v2…