PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We Ready For An Agent-Native Memory System?

Wei Zhou, Xuanhe Zhou, Shaokun Han, Hongming Xu, Guoliang Li, Zhiyu Li, Feiyu Xiong, Fan Wu

arXiv:2606.24775v1cs.CLcs.DBcs.IR

TL;DR

Existing agent-memory evaluations provide limited system-level evidence because they often treat memory as a black box and focus on end-to-end task metrics. This paper decomposes memory into core modules and evaluates representative systems across workloads and technical variants, finding that no architecture dominates universally and that cost-performance trade-offs depend on workload-aligned design.

  • Problem

    Prior evaluations inadequately compare representative memory architectures and system-level concerns such as operational cost, architectural trade-offs, and robustness under dynamic updates.

  • Method

    The paper introduces a four-module data-management framework and evaluates representative memory systems through unified end-to-end workloads and controlled component variants.

  • Results

    No single memory architecture dominates all scenarios; hybrid systems lead conversational QA, while graph-based methods excel at single-hop factual recall but struggle with temporal reasoning.

  • Takeaways & Limitations

    Memory effectiveness depends on aligning memory structure with workload bottlenecks, while localized maintenance is more cost-efficient than global reorganization.

Abstract

from arXiv · show

Memory for large language model (LLM) agents has rapidly evolved from simple retrieval-augmented mechanisms into a data management system that supports persistent information storage, retrieval, update, consolidation, and dynamic lifecycle governance throughout agent execution. Despite this evolution, existing evaluations still benchmark agent memory mainly through end-to-end task success metrics (e.g., F1, BLEU), while treating the underlying system as a monolithic black box. As a result, critical system-level concerns, including operational costs, architectural trade-offs across memory modules, and robustness under dynamic knowledge updates, remain insufficiently explored. In this paper, we present a systematic experimental study of agent memory from a data management perspective. We propose an analytical framework that decomposes agent memory into four core modules: memory representation and storage, extraction, retrieval and routing, and maintenance. Under this framework, we evaluate 12 representative memory systems and two reference baselines across five benchmark workloads spanning 11 datasets. Our extensive end-to-end evaluation shows that no single architecture dominates across all scenarios; instead, effectiveness depends heavily on how well the memory structure aligns with the workload bottleneck. Furthermore, through fine-grained ablation studies, we quantify their individual effects on representation fidelity, retrieval precision, update correctness, and long-horizon stability. Finally, we reveal cost-performance trade-offs under realistic workloads, showing localized maintenance is more cost-efficient than global reorganization. Based on these findings, we identify promising directions towards building truly agent-native memory systems. The code is publicly available at https://github.com/OpenDataBox/MemoryData.

1 Introduction

Agent memory has evolved into a persistent data-management layer for long-horizon execution, but prior evaluations inadequately compare architectures and system-level trade-offs. This paper decomposes memory into four modules and evaluates representative systems across diverse workloads to characterize effectiveness, robustness, and cost.

  • Motivation: Agent memory stores information beyond a single inference step and writes, updates, indexes, and routes context back into the reasoning loop.Poor architecture can cause factual contradictions, catastrophic forgetting, or unacceptable latency during continuous execution.
  • Research gap: Prior benchmarks show external memory helps factual recall and long-context tasks but omit representative architectures and complex agentic execution scenarios.These omissions make principled cross-system comparisons difficult.
  • Framework: The paper decomposes agent memory into representation and storage, extraction, retrieval and routing, and maintenance.The taxonomy organizes existing approaches by underlying design principles for principled comparison.
  • Evaluation: The evaluation covers task effectiveness, retrieval fidelity, dynamic update robustness, long-horizon stability, and operational cost.These perspectives treat memory as a data-management system rather than a monolithic task component.
  • Findings: No single memory architecture dominates all scenarios: hybrid systems lead conversational QA, while graph methods excel at single-hop factual recall but struggle with temporal reasoning.The findings also report robustness across LLM backbone variants because systems externalize evidence localization before answer generation.

2 Preliminaries

The paper frames agent memory as a persistent, updatable data-management object rather than merely an algorithmic component, distinguishing it from RAG and context engineering. It also characterizes memory by temporal and functional types and emphasizes lifecycle management under evolving workloads.

  • Perspective: The paper studies agent memory as a standalone data-management object, contrasting with views that treat it primarily as an algorithmic component of an LLM or agent pipeline.The scope covers information updated and maintained under real agent workloads.
  • Definitions: Agent memory is a persistent data-management object that maintains cumulative state beyond one inference step and makes it accessible during future reasoning and action.Stored information can include dialogue history, tool logs, distilled facts, and user preferences.
  • Definitions: Memory types span short-term versus long-term information temporally, and episodic, semantic, procedural, and preference information functionally.Long-term memory persists across sessions and includes concrete events, abstract facts, reusable strategies, and user preferences.
  • Scope distinctions: Unlike read-only RAG, agent memory persists and updates agent-specific state while governing representation, storage, retrieval, and maintenance across the long-term lifecycle.RAG fetches relevant passages from a static corpus to augment a single generation step.
  • Workload characteristics: Agent-memory queries are often semantic and contents evolve through uncertain, partial, and potentially conflicting observations.These workloads rely on approximate matching, query rewriting, or LLM-guided retrieval rather than only exact predicates over rigid schemas.

3 Method Overview

The method organizes agent memory around a unified taxonomy of representation, storage, extraction, retrieval, and routing choices. These components determine how raw interaction data becomes persistent, searchable memory for agent execution.

  • Framework: The taxonomy analyzes existing agent memory systems across four components and summarizes representative methods under a unified framework.The four components are representation and storage, extraction, retrieval and routing, and maintenance.
  • Representation and storage: Logical representation defines structural encoding and organization, while physical storage determines persistence and indexing structures.Together they shape capacity, accessibility, expressiveness, retrieval granularity, and downstream reasoning compatibility.
  • Logical representation: Representation methods organize memory into models such as graphs or vector spaces, affecting how efficiently systems search, combine, and use historical context.The taxonomy includes token-level sequence representations and temporal knowledge graphs.
  • Extraction: Memory extraction transforms heterogeneous inputs such as dialogues and tool logs into logical memory primitives before persistence.Raw sequence concatenation minimizes computational overhead by bypassing explicit extraction prompts.

PREFERS

The example encodes a preference as a structured relation between an entity and vegetarian food.

  • Structured preference: The memory representation records Kim’s preference for vegetarian food as a structured relation.The displayed relation uses the predicate PREFERS.

LIVES_IN

This section presents memory representation, storage, and retrieval designs, ranging from structured memory objects and vector indexes to topological traversal and agentic routing.

  • Representation: MemOS organizes plain-text, activation, and parametric memory in a unified MemCube with structured details.
  • Storage: Memory storage determines capacity limits, access speed, throughput, and scalability through caches, files, vector engines, or databases.
  • Retrieval: Retrieval methods include native attention, dense semantic search, topological subgraph traversal, and autonomous agentic routing.
  • Retrieval: Memory items can be stored as text snippets with associated retrieval scores in indexed representations.

KEEPS_PET

The passage identifies Charlie’s residence and mentions a pet-related context.

  • Charlie’s residence is referenced.
  • The passage includes an ellipsis after the residence reference.

LIVES_IN

This section describes retrieval and routing as the process of finding relevant historical context, alongside extraction approaches that transform interaction traces into logical memory structures.

  • Extraction: Memory extraction converts heterogeneous inputs such as dialogues and tool logs into logical memory primitives before persistence.
  • Extraction: Extraction may use raw sequence concatenation, structured schemas, or predefined fields to make stored information predictable.
  • Retrieval and routing: Memory retrieval and routing identify relevant historical context to inform the agent’s current reasoning state.

C2 C3 C4

This section characterizes memory maintenance through eviction, semantic consolidation, CRUD operations, and retrieval-oriented routing strategies that govern updates and memory growth.

  • Eviction: Capacity-driven eviction includes constraint-based hard eviction and score-based priority eviction.
  • Consolidation: LLM-driven semantic consolidation fuses historical states with new payloads through summarization across parent nodes.
  • CRUD execution: Tool-driven CRUD execution applies explicit Create, Read, Update, or Delete commands through LLM-guided interfaces.
  • Retrieval maintenance: Maintenance-related retrieval designs include generative query expansion, multi-stage hybrid execution, sequential routing, and parallel ensemble retrieval.
  • Maintenance: Memory maintenance updates, compresses, forgets, and removes information over time.
  • Versioning: Timestamp-based multi-versioning preserves historical continuity by logically deprecating expired or conflicting facts rather than physically deleting rows.

4 End-to-End Assessment

Across five benchmark workloads, memory effectiveness depends on matching memory organization and routing to the workload bottleneck rather than using one universal architecture. Retrieval fidelity, update robustness, long-horizon stability, and operational efficiency likewise depend on evidence organization and maintenance scope.

  • Overall Effectiveness: No single memory architecture dominates: relation- and time-aware retrieval suits dispersed cross-session reasoning, coarse-to-fine filtering suits coherent long dialogues, and trace preservation suits stateful execution.MemoryOS and MemOS remain close to the overall frontier, while workload-specific bottlenecks determine which organization is effective.
  • Retrieval Fidelity: 39.0 Recall@1 is achieved by SimpleMem, while A-MEM and MemTree reach 69.5/85.9 and 59.7/80.5 on Recall@5/@10, respectively.Structured systems remain more stable as the evidence distance gap increases, whereas flat Embedding RAG drops sharply after the shortest-gap bin.
  • Update Robustness: Reliable post-update behavior requires revisable representations, workload-matched query selectivity, and stronger LLM backbones applied after grounding rather than as the primary stale-memory resolver.External organization produces a more stable evidence set across LLM backbones, while LLM-side synthesis produces greater cross-backbone movement.
  • Long-Horizon Stability: As the memory horizon grows, multi-view filtering, relation-aware indexing, and coarse-to-fine summarization address distractors, distant evidence, and session-level narrowing, respectively.Pure long-context prompting and flat dense memory degrade more sharply as the effective horizon increases.
  • Operational Cost: 48.3 Normalized Utility at 3.67 s Avg. Operation Latency/Query places LightMem on the efficiency frontier, while MemoryOS reaches 82.0 utility at 28.6 s and Cognee and Zep exceed 84 utility only after 116.5 s and 155.1 s.Efficiency is governed by maintenance scope: localized updates and search are cheaper, whereas broad recomputation and whole-memory coordination drive latency.

5 Fine-Grained Component Comparison

The component ablations show that agent-memory quality depends on preserving recoverable evidence, structuring retrieval selectively, and consolidating updates conservatively rather than maximizing abstraction or complexity.

  • Memory Representation and Storage: Preserving original conversational content supports factual recall and reasoning better than stronger abstraction or hierarchy.LightMem User-Only Raw performs best across the reported metrics, while deeper organization offers only modest gains.
  • Memory Extraction: Coverage-preserving extraction provides a more stable balance between factual retrieval and downstream reasoning than aggressive filtering.The variants show that broader extraction can improve reasoning or retrieval without uniformly improving both.
  • Memory Retrieval and Routing: 24.6 Ans. F1 and 27.5 Substr. EM are achieved by A-MEM with Hybrid-Balanced fusion, outperforming its Hybrid Sparse-Leaning variant.SimpleMem likewise performs best with Planning Only, while adding reflection does not improve the reported retrieval outcomes.
  • Memory Retrieval and Routing: Moderate fusion and lightweight planning improve retrieval effectiveness, whereas extra reflection adds overhead without further gains.These findings favor targeted retrieval structure over simply increasing sparse matching or reasoning steps.
  • Memory Maintenance: Conservative consolidation better preserves updated facts and long-horizon consistency than delayed flushing or overly coarse summarization.Delayed flushing leaves recent evidence fragmented, while coarse summaries obscure sparse but useful cues.

6 Conclusion

The paper reviews agent memory from a data-management perspective, evaluates systems and individual modules, and identifies suitable architectures, influential factors, and research directions.

  • Conclusion: The study evaluates representative agent-memory systems end to end and examines their suitable application scenarios.It also constructs module variants to analyze representation, extraction, routing, maintenance, operational costs, and long-horizon stability.
  • Conclusion: The authors summarize findings, provide guidance for selecting memory architectures, and outline promising research directions.They also release the testbed and evaluation framework.
Loading 2606.24775v1…