Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

Dongming Jiang; Yi Li; Songtao Wei; Jinxin Yang; Ayushi Kishore; Alysa Zhao; Dingyi Kang; Xu Hu; Feng Chen; Qiannan Li; Bingzhe Li

Source-linked AI summary

Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

Dongming Jiang, Yi Li, Songtao Wei, Jinxin Yang, Ayushi Kishore, Alysa Zhao, Dingyi Kang, Xu Hu, Feng Chen, Qiannan Li, Bingzhe Li

arXiv:2602.19320v2Feb 22, 2026cs.CLcs.AI

TL;DR

Agentic memory addresses the challenge of maintaining persistent state for long-horizon LLM-agent interactions beyond fixed context windows, amid limited empirical evidence. This survey develops a four-structure taxonomy and analyzes evaluation validity, benchmark scalability, and system bottlenecks. It finds that lexical metrics can diverge from semantic judgments, while semantic-judge rankings are more stable, and concludes that broader standardized evaluation is needed.

Problem
Fixed context windows constrain persistent state, while current evaluations may be underscaled, semantically misaligned, and inattentive to system costs.
Method
The survey introduces a four-structure taxonomy and systematically analyzes benchmark saturation, metric validity, backbone sensitivity, and memory-maintenance costs.
Results
Lexical metrics systematically diverge from semantic judgments, whereas semantic-judge architecture rankings remain consistent across grading rubrics and better reflect underlying memory quality.
Takeaways & Limitations
Reliable agentic-memory evaluation requires attention to benchmark structure, prompt design, semantic validity, and system-level scalability.
Takeaways & Limitations
The empirical analysis covers representative architectures and selected benchmarks, so results may vary with implementations, prompts, model versions, and API behavior.

Abstract

from arXiv · show

Agentic memory systems enable large language model (LLM) agents to maintain state across long interactions, supporting long-horizon reasoning and personalization beyond fixed context windows. Despite rapid architectural development, the empirical foundations of these systems remain fragile: existing benchmarks are often underscaled, evaluation metrics are misaligned with semantic utility, performance varies significantly across backbone models, and system-level costs are frequently overlooked. This survey presents a structured analysis of agentic memory from both architectural and system perspectives. We first introduce a concise taxonomy of MAG systems based on four memory structures. Then, we analyze key pain points limiting current systems, including benchmark saturation effects, metric validity and judge sensitivity, backbone-dependent accuracy, and the latency and throughput overhead introduced by memory maintenance. By connecting the memory structure to empirical limitations, this survey clarifies why current agentic memory systems often underperform their theoretical promise and outlines directions for more reliable evaluation and scalable system design.

1 Introduction

Agentic memory extends LLM agents beyond fixed context windows, but its empirical foundations remain limited by inadequate evaluation and overlooked system costs. This survey responds with a structure-oriented taxonomy and analyses linking memory designs to reliability and scalability bottlenecks.

Fixed context windows limit agents’ ability to retain and manipulate persistent state across long-horizon interactions.
Benchmarks are often underscaled, metrics emphasize surface overlap over semantic utility, and latency and throughput costs are rarely measured.
The survey organizes Memory-Augmented Generation around four memory structures and uses this taxonomy to analyze system behavior.
Its analyses examine benchmark saturation, metric misalignment, judge prompt sensitivity, backbone dependence, retrieval latency, update overhead, and throughput degradation.
By linking memory structures to empirical limitations, the survey provides a diagnostic framework for understanding effectiveness, failure, trade-offs, and scalable system design.
Compared with prior surveys focused mainly on theory, this work bridges theory and practice through structure-oriented analysis and comprehensive benchmark evaluations.

2 Background

Agentic memory gives LLM agents persistent, writable state that evolves across interactions. Its behavior is governed by explicit retrieval and read–write operations over external memory rather than updates to model weights.

At each step, the agent conditions its output and state on observations and an external memory state.
The retrieval operator ψ(M_t; q_t) supplies memory given a query, and integration combines retrieved information with the agent’s processing.
Unlike parametric learning, the mechanism changes behavior through explicit read–write operations over evolving memory rather than by modifying model weights.
Agentic memory couples inference-time recall with subsequent writing, updating, or consolidation to maintain a useful long-term state.

3 Taxonomy of Agentic Memory

The taxonomy classifies contemporary Memory-Augmented Generation systems by four primary memory structures: lightweight semantic, entity-centric and personalized, episodic and reflective, and structured and hierarchical. These structures differ in organization, temporal abstraction, relational encoding, and maintenance demands.

Lightweight Semantic Memory: Lightweight semantic memory stores largely independent textual units in vector space and retrieves them by top-k similarity search.
Lightweight Semantic Memory: Semantic compression and context-management methods select, summarize, or encode information to reduce context burden with minimal overhead.
Lightweight Semantic Memory: Lightweight semantic representations suit short- to medium-horizon recall but are limited for precise state tracking and long-term reasoning.
Entity-Centric and Personalized Memory: Entity-centric and personalized memory organizes persistent information around explicit entities, users, tasks, or preferences using structured records or attribute–value pairs.
Episodic and Reflective Memory: Episodic and reflective memory organizes interactions into episodes or higher-level summaries and periodically consolidates salient experience.
Structured and Hierarchical Memory: Structured and hierarchical systems partition memory into tiers or encode relationships among elements through graphs and other relational representations.
Structured and Hierarchical Memory: These systems enable adaptive management but introduce greater complexity and nontrivial maintenance overhead.
Discussion: The four-category structure provides a foundation for analyzing how memory organization influences accuracy, efficiency, and reliability.

4 Evaluation and Pain Points

The survey identifies four practical bottlenecks in agentic-memory evaluation and deployment: benchmark saturation, metric misalignment, backbone-sensitive memory operations, and system-level latency and cost. Across these dimensions, it argues that memory benefits require structurally demanding tests, semantically valid evaluation, reliable backbones, and scalable maintenance.

Evaluation and Pain Points: The analysis examines benchmark validity, metric reliability, system efficiency, and backbone sensitivity as four constraints on robust agentic-memory deployment.These dimensions ask whether evaluations test memory rather than context length, capture semantic coherence, account for latency and cost, and expose silent memory-operation failures.
Benchmark Validity: Only datasets substantially exceeding the active context window, such as LongMemEval-M at >1M tokens, structurally require external memory.HotpotQA, MemBench, and LoCoMo remain within a 128k window, creating high theoretical saturation risk.
Benchmark Validity: Saturation risk depends on whether benchmark volume, interaction depth, and entity diversity exceed long-context models’ representational capacity, not on surface difficulty alone.Multi-session and higher-diversity settings increase temporal dependency, interference, and relational reasoning demands, but may remain solvable in context when dependencies fit within the active window.
Benchmark Validity: The Context Saturation Gap, ∆ = Score_MAG − Score_FullContext, diagnoses memory benefit beyond a brute-force full-context baseline under matched evaluation conditions.A large positive ∆ is especially informative in out-of-window or lost-in-the-middle regimes, but the measure is diagnostic rather than a strict pass/fail criterion.
Metric Reliability: Lexical metrics can diverge from semantic utility: AMem has semantic Rank 4 but F1 Rank 5 with Score 0.116, while SimpleMem has F1 0.268 despite semantic score < 0.30.The reported Paraphrase Penalty and Negation Trap explain how overlap metrics can penalize correct abstractions or mask factual errors.
Metric Reliability: Semantic-judge rankings remain consistent across rubrics and prompt formulations, supporting LLM-as-a-judge while underscoring careful prompt design.Absolute scores vary with prompt strictness, but relative architecture ordering remains robust.
Backbone Sensitivity: Backbone choice affects memory reliability: Qwen-2.5-3B shows more format errors and lower end-task scores than gpt-4o-mini, with corruption arising from failed writes.Append-only systems are comparatively robust, whereas graph-based and episodic architectures are more sensitive to structured-generation demands.
System Efficiency: Full Context has generation latency Tgen ≈1.73s, while SimpleMem and LOCOMO achieve sub-second latency below 1.1s and MAGMA maintains approximately 1.46s.The comparison measures user-perceived latency as Tread + Tgen and highlights the trade-off between retrieval overhead and large-prompt prefill cost.

5 Conclusion and Future Directions

Agentic memory is constrained by evaluation validity, scalability, and robustness, not architecture alone.

Agentic memory systems face limitations spanning evaluation validity, scalability, and robustness.

1. Rethinking Benchmark and Evaluation De-

Current benchmarks and metrics can obscure the value of external memory by saturating context capacity and emphasizing lexical rather than semantic correctness. Evaluation also requires calibrated, robust use of LLM judges.

Benchmarks should stress task volume, temporal depth, entity diversity, and long-range dependency as context windows expand.The Context Saturation Gap (∆) is proposed as a diagnostic signal.
F1-style metrics can miss semantic correctness, while LLM-as-a-judge requires prompt calibration and robustness checks.

2. Designing Scalable and Robust Agentic Mem-

Scalable agentic memory requires balancing accuracy, latency, cost, and reliability because memory structure affects both reasoning capability and maintenance overhead.

Structured memory can improve reasoning but introduces maintenance overhead, whereas lightweight approaches improve efficiency but may lack abstraction.
Future systems should explicitly model write latency, maintenance throughput, and user-facing cost.
Backbone-aware operations, constrained decoding, validation layers, and adaptive schemas can reduce silent corruption.

Limitations

The paper frames agentic memory through external recall and update processes, structured taxonomies, related-work comparisons, and explicit limitations of its empirical scope.

Limitations: The taxonomy may miss recent systems, and empirical findings are limited to representative architectures and selected benchmarks whose results may vary across implementations and model settings.
B.1 Memory Operations in Agentic Systems: Agentic memory is an external non-parametric subsystem combining inference-time recall with memory updates.It reads, writes, consolidates, and forgets information to maintain a long-term store.
B.1 Memory Operations in Agentic Systems: Retrieved memory is formatted and integrated with current context through concatenation, schema-based slots, or cross-modal fusion.
B.1 Memory Operations in Agentic Systems: Retrieval ranks memory using dense similarity, sparse matching, or reranking, but ideal relevance is conditional on downstream decisions.
B.1 Memory Operations in Agentic Systems: Memory updates can use actions such as STORE, UPDATE, SUMMARIZE, LINK, EVICT, or DELETE, with rule-based, model-driven, or learned policies.
C Related Work: The survey differs from primarily theoretical surveys by emphasizing structure-oriented analysis, management and optimization strategies, and evaluations across multiple benchmarks.

D Prompt Library

The prompt library organizes agentic-memory evaluation into memory construction, query processing, answer generation, and multiple judge protocols that vary in semantic tolerance and scoring.

D Prompt Library: Prompt designs are organized into Memory Construction, Query Processing, and Response Generation stages across evaluated systems.Table 6 summarizes operational patterns across memory architectures.
D.1 Memory Construction and Retrieval: Memory construction consolidates raw interaction history into long-term storage, using profile-based, episodic, or graph-based representations.MemoryOS extracts evolving user traits, Nemori segments dialogue into episodes, and MAGMA builds relational structures.
D.1 Memory Construction and Retrieval: The build-prompt specification is intended to summarize interactions into atomic facts for memory consolidation.The supplied example describes summarizing an interaction into atomic facts.
D.1 Memory Construction and Retrieval: Query refinement is bypassed by Nemori and MemoryOS but used by AMem and SimpleMem to optimize search vectors or keywords.The systems differ between direct semantic search and LLM-based query transformation.
D.2 Response Generation: Answer-generator prompts are shared across RAG, MAG, and Full-Context baselines but adapt to context strategy and constraints such as temporal awareness or multi-hop reasoning.Examples include profile-enriched role-playing for MemoryOS and episodic retrieval for Nemori.
D.3 LLM-as-a-Judge Evaluation Protocols: Evaluation uses gpt-4o-mini as the backbone judge with literature-derived correctness prompts and sensitivity rubrics.The protocols are designed to evaluate diverse grading criteria across architectures.
D.3.1 Literature-Derived Baselines: The correctness baseline accepts semantically equivalent answers, paraphrases, wording differences, and alternate time formats as CORRECT.It compares a question, gold answer, and generated answer using a binary JSON label.

E Baseline Configurations

Baseline configurations standardize retrieval and generation settings for fair comparison while preserving architecture-specific capacity defaults.

E Baseline Configurations: Evaluated systems follow repository defaults with standardized modifications applied across baseline architectures.The modifications target embeddings, temperature, retrieval scope, and token settings.
E Baseline Configurations: All dense retrieval uses all-MiniLM-L6-v2, replacing system-specific embedding models for controlled semantic matching.This standardizes the embedding component across systems.
E Baseline Configurations: Generation temperature is fixed at 0.3 across backbone LLMs to balance determinism and reasoning capability.The same temperature is used for all evaluated backbones.
E Baseline Configurations: Final answer synthesis retrieves top-k = 10 raw-history or memory chunks when retrieval is used.The retrieval scope is uniform across systems.
E Baseline Configurations: Maximum generation and context tokens remain at repository-specific defaults to preserve each architecture’s intended design.Structural capacity is therefore not fully standardized.
E Baseline Configurations: Table 7 reports unified hyperparameters together with each system’s operational parameters.The configurations are summarized for comparison across evaluated memory architectures.

F Case Studies: Why Lexical Metrics Fail

The case studies identify recurring ways lexical metrics diverge from semantic judgments, showing that token overlap can misrepresent both correct reasoning and factual accuracy.

F Case Studies: Why Lexical Metrics Fail: The qualitative analysis examines recurring mechanisms behind disagreements between lexical metrics such as F1 and semantic judgments.The cases target inherent limitations of token-level evaluation rather than isolated examples.
F Case Studies: Why Lexical Metrics Fail: Four failure patterns are identified: surface variation, semantic equivalence gaps, polarity flips, and entity drift.These mechanisms cover both penalized correct answers and inflated scores for incorrect answers.
F Case Studies: Why Lexical Metrics Fail: Surface variation penalizes correct answers that use additional context or alternative phrasing because lexical overlap decreases.The semantic content can remain correct despite lower token similarity.
F Case Studies: Why Lexical Metrics Fail: Semantic equivalence gaps assign zero or near-zero scores to equivalent meanings expressed through different formats or synonyms.This reflects sensitivity to representation rather than underlying meaning.
F Case Studies: Why Lexical Metrics Fail: Polarity flips preserve high token overlap even when minor lexical changes, such as negation, reverse the meaning.High lexical similarity can therefore coexist with semantic contradiction.
F Case Studies: Why Lexical Metrics Fail: Entity drift produces inflated lexical similarity when incorrect entities or values replace correct ones within similar sentence structures.This failure mode directly concerns factual errors masked by structural overlap.
F Case Studies: Why Lexical Metrics Fail: The cases show lexical metrics are systematically misaligned with abstraction, normalization, and compositional reasoning in modern LLM systems.Traditional token-based evaluation may fail to reflect or may penalize improvements in reasoning quality.