Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

1 Introduction

• Existing agent evaluations emphasize reasoning and execution, leaving memory—storing, updating, and retrieving information—under-explored.The paper defines memory agents as agents equipped with memory mechanisms, including parameters, vectors, textual histories, or external databases; it focuses primarily on textual histories and external databases.Source: §1 · pp. 1–2
• Earlier benchmarks use short contexts or static long-context inputs, while memory agents incrementally process and consolidate information.LongMemEval addresses incremental input through session-by-session synthetic conversations, but the paper describes its topical diversity and interaction realism as limited.Source: §1 · pp. 2–3
• MemoryAgentBench evaluates memory agents across accurate retrieval, test-time learning, long-range understanding, and selective forgetting.The benchmark presents information incrementally through multi-turn interactions and combines reconstructed datasets with two newly created datasets.Source: §1 · pp. 2–3
• Existing memory agents can perform well on some tasks but still face significant challenges in other aspects of memory.The benchmark evaluates simple agents, commercial agents, and systems with diverse memory mechanisms.Source: §1 · p. 3

2.1 Benchmarks on Long-Context and Memory

• Long-context benchmarks assess extensive information in a single pass, while RAG benchmarks typically assume static knowledge and short-lived interactions.Existing memory-agent benchmarks also have limitations in conversation length, topical diversity, or coverage of the four competencies.Source: §2.1 · pp. 3–4
• Table 1 compares benchmark question counts, context depth, competency coverage, and evaluation across long-context, RAG, and agentic-memory methods.The table caption defines context depth as the number of tokens in the history and notes that some values are approximations or unreported.Source: Table 1 · p. 4

2.2 Agents with Memory Mechanisms

• Memory agents use context buffers, retrieval-augmented memory, or iterative agentic loops to store and use past information.RAG variants include string matching, embedding-based retrieval, and structured representations; agentic systems can reformulate queries, retrieve evidence, reflect, and iterate.Source: §2.2 · pp. 4–5

3.1 Dataset Preperation

• Accurate Retrieval datasets include document question answering and long-dialogue retrieval, alongside EventQA for recalling and reasoning about temporal event sequences.LongMemEval (S*) reformulates chat history into five dialogues of about 355K tokens with 300 questions; EventQA selects the correct event after up to five prior events.Source: §3.1 · pp. 5–6
• Test-Time Learning datasets assess classification from labeled examples seen in context and movie recommendations based on dialogue history.The classification tasks reconstruct five datasets; the recommendation task asks agents to recommend twenty movies after thousands of dialogue turns.Source: §3.1 · p. 6
• Long-Range Understanding is assessed through novel summarization and questions requiring reasoning across longer narrative ranges.The summarization task requests 1,000–1,200 words, while Detective QA includes ten novels and 71 questions.Source: §3.1 · p. 6
• FactConsolidation tests selective forgetting with ordered contradictory facts and questions answered from the final memory state.Its contexts range from 6K to 262K tokens and include single-hop and multi-hop questions.Source: §3.1 · p. 6

3.2 Different Categories of Memory Agents

• The evaluated agent categories are long-context agents, RAG agents, and agentic memory agents.The formulation describes context buffering, external-memory retrieval variants, and iterative reasoning cycles involving memory lookups and working-memory updates.Source: §3.2 · pp. 6–7

3.3 Datasets and Agents Formulation

• Each dataset is delivered as sequential chunks with memorization instructions, followed by questions; agents incrementally absorb chunks before answering.Multiple questions can follow one context, and standardized prompt templates are used within each evaluation category; selective forgetting prompts direct agents to prioritize newer facts.Source: §3.3 · p. 7

4.1 Experimental Setup

• The evaluation includes long-context, RAG, and agentic-memory agents, with RAG further divided into simple, embedding-based, and structure-augmented systems.Chunk size is 512 for specified long-text AR tasks and all SF tasks, and 4096 for other tasks; commercial memory agents uniformly use 4096.Source: §4.1 · pp. 7–8

4.2 Overall Performance Comparison

• Across competencies, RAG agents usually outperform GPT-4o-mini on Accurate Retrieval, long-context models lead on Test-Time Learning and Long-Range Understanding, and all methods struggle with multi-hop forgetting.On multi-hop Selective Forgetting, methods reach at most 7% accuracy; only long-context agents perform fairly reasonably on single-hop cases.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 Analysis and Ablation Study

• The analysis varies input chunk size, retrieval top-k, backbone model, and dataset validation, with additional latency, context-length, and cost-performance results in appendices.Source: §4.3 · pp. 8–9

4.3.1 Ablation Study on Input Chunk Size

• Smaller chunks can improve Accurate Retrieval when resources permit, but varying chunk size hurts Long-Range Understanding.The authors attribute the retrieval benefit to more relevant fine-grained segments and the LRU decline to RAG's difficulty integrating a large coherent context.Source: §4.3.1 · p. 9, Figure 2 · p. 9

4.3.2 Ablation Study on Retrieval TopK

• Increasing retrieved chunks generally improves performance across most tasks, while 10 chunks of 4096 tokens already yield approximately 40k input tokens.The study omits 20-chunk settings because of the high token volume and its demands on model capacity.Source: §4.3.2 · p. 9, Figure 3 · p. 9

4.3.3 Ablation Study on Backbone Model

• For RAG agents, a sufficiently strong backbone is no longer the main bottleneck, whereas a stronger backbone substantially improves MIRIX.Upgrading RAG to GPT-4.1-mini yields only marginal improvements compared with the default setup.Source: §4.3.3 · p. 10, Table 4 · p. 10

4.3.4 Validation of Dataset FactConsolidation

• FactConsolidation is solvable in short contexts, but performance falls as context grows, including o4-mini’s FactCon-MH score dropping from 80.0 at 6K to 14.0 at 32K.The authors conclude current memory agents lack strong long-range reasoning for longer historical inputs.Source: §4.3.4 · p. 10

• MemoryAgentBench assesses accurate retrieval, test-time learning, long-range understanding, and selective forgetting across memory agents.It restructures existing datasets and adds EventQA and FactConsolidation under a consistent protocol spanning long-context, RAG-based, and commercial agents.Source: §5 · p. 10
• Current agents show substantial limitations on tasks requiring dynamic memory updates and long-range consistency.Source: §5 · p. 10
• Budget constraints limited experiments to representative memory agents, and the authors plan to evaluate more agents in future work.Source: §5 · p. 10

• The authors state that their work follows the ICLR Code of Ethics and associated author guidance, with potential impacts and mitigations documented.Source: Ethics Statement · p. 11
• The evaluation uses dialogs and license-compliant corpora, with no personally identifiable information or data from minors collected.Source: Ethics Statement · p. 11
• Only safety-screened prompts will be released, with usage notes discouraging surveillance-oriented applications; code and benchmark artifacts have stated licenses.Code is planned under MIT and datasets and benchmark artifacts under CC BY 4.0, while third-party materials retain their original licenses.Source: Ethics Statement · p. 11

Reproducibility Statement

• Upon acceptance, the authors plan to release code, data, prompts, scripts, run recipes, pinned dependencies, a containerized environment, and hardware and software details.Source: Reproducibility Statement · p. 11

A The Use of Large Language Models (LLMs)

• An LLM assisted with polishing text and generating character icons used in the main plot visualization.Source: A The Use of Large Language Models… · p. 18

B Details of Dataset

• Dataset documentation covers curation, metrics, average context length, and brief descriptions, with details in Table 2.Source: B Details of Dataset · p. 18

B.1.1 Definition of AR

• Accurate Retrieval means identifying and retrieving important information dispersed across a long dialogue history.Source: §B.1.1 · p. 18

B.1.2 Details on AR datasets

• AR datasets include reconstructed QA contexts of 197K or 421K tokens, LongMemEval histories of about 355K tokens, and EventQA continuation questions built from five books.The QA answers use substring exact match; LongMemEval uses GPT-4o judging, while EventQA reports accuracy.Source: §B.1.2 · pp. 18–19

B.2.1 Definition of TTL

• TTL measures learning new tasks from conversation history; its datasets test classification over labeled examples and movie recommendation from prior dialogues.The evaluation includes multi-class classification and Recall@5 for recommendation.Source: §B.2.1 · p. 19, §B.2.2 · p. 19

B.3.1 Definition of LRU

• LRU evaluates high-level understanding of long-form input, while SF tests detecting contradictions and discarding outdated facts; LRU uses En.Sum and SF uses single- and multi-hop FactConsolidation.The SF datasets order outdated information before newer information and use SubEM for QA.Source: §B.3.1 · p. 19, §B.3.2 · p. 19, §B.4.1 · p. 20

B.5 Justification for competencies based on cognitive science

• The four competencies—accurate retrieval, test-time learning, long-range understanding, and selective forgetting—reflect distinct memory dimensions including recall, consolidation, and interference handling.The authors connect these dimensions to cognitive-science accounts of memory.Source: §B.5 · p. 20

C Detailed Memory Agents Description

• The evaluated systems span long-context models, lexical and embedding RAG, structure-augmented RAG, and agentic memory systems; prompts and settings are standardized across systems.Examples include MemGPT’s hierarchical memory, MIRIX’s specialized memory types, and prompt instructions for memory construction and query execution.Source: §C.1 · p. 21, §C.2 · pp. 21–22, §C.3 · p. 22

E.1 Detailed Results on TTL

• RAG-based agents generally underperform their GPT-4o-mini backbones across all three TTL task types.The detailed results are reported in Table 7.Source: §E.1 · p. 22, Table 7 · p. 25

E.2 Results on Input Chunk Size Ablation Study

• The appendix reports chunk-size and retrieval-TopK ablations, and provides example prompts for memory construction and task execution.The reported chunk-size sets are {512, 4096} and {512, 1024, 2048, 4096}; retrieval TopK ranges from 2 to 10.Source: §D.1 · p. 22, §D.2 · p. 22, §E.2 · p. 23

E.4 Results on Different Context Length Ablation Study

• As input length grows, long-context agents’ AR performance declines after generally satisfactory results around 50K tokens; Mem0 and Cognee trail GPT-4o-mini even at shorter lengths.The context-length comparison is reported in Table 10.Source: §E.4 · p. 24, Table 10 · p. 25

E.5 Results on Computational Latency Analysis

• Smaller chunks require substantially more memory-construction time, and Mem0, Cognee, and MIRIX require especially high resources during construction.The latency analysis reports construction and query execution for MH-QA and LongMemEval (S*).Source: §E.5 · pp. 25–26, Table 12 · p. 26

E.6 GPU Memory Usage Comparison

• The GPU-memory comparison reports peak embedding-model usage on MH-QA using a single A100 80GB GPU; the settings section specifies embedding choices and chunk sizes by agent and task.Mem0, Zep, Cognee, and MIRIX use 4096-token chunks across datasets because their construction is more time- and API-cost-intensive.Source: §E.6 · p. 26, Table 13 · p. 26, §F.2 · p. 27

F.1 Max Output Tokens

• The experimental settings also provide task-specific maximum output-token limits.These limits are listed in Table 14.Source: §F.1 · p. 27, Table 14 · p. 27

G Task Rationale and Justification for Selective Forgetting Task

• The selective-forgetting task uses a controlled synthetic setting because naturalistic long histories with precise forgetting annotations are difficult to construct.The authors describe the task as a proxy for evaluating removal of outdated, redundant, or superseded information.Source: G Task Rationale and Justification for… · p. 28

H.1 Terminology and Design Rationale

• TTL distinguishes rule and preference learning from static fact retrieval, requiring generalization from sequential examples to unseen inputs without target-task pretraining or fine-tuning.The paper frames the protocol as a standardized, reproducible operationalization of in-situ learning.Source: §H.1 · pp. 28–29
• The TTL protocol separates incremental acquisition of labeled examples from testing on held-out inputs, reducing confounds from dynamic feedback loops.The framework is intended to isolate memory-enabled learning and is presented as a simplified operationalization to be extended to fully interleaved online learning.Source: §H.1 · p. 29

H.2 Zero-Shot Baseline Validation Experiments

• The zero-shot baselines omit the historical example sequence, preventing test-time learning; GPT-4o-mini’s full-memory result is included as the direct comparison.Source: §H.2 · p. 29
• Across both test-time learning (TTL) tasks, models average below 4% accuracy zero-shot, while GPT-4o-mini reaches 48.6% with the full historical example sequence, a 45.2 percentage-point absolute improvement.Source: §H.2 · p. 29

I.1 Methodology

• The comparison reports amortized inference cost per query and performance across four datasets, assuming context caching for long-context systems and excluding one-time embedding-indexing costs.The cached input discounts are 50% for GPT-4o-mini and 75% for GPT-4.1-mini.Source: §I.2 · p. 30, §I.1 · p. 30, Table 17 · p. 30

I.3 Key Insights

• BM25 costs less than $0.001–$0.006 per query but scores 52.1 on Detective QA versus 63.4 for Long-Context models.The authors identify this as an efficiency–reasoning trade-off for tasks requiring global reasoning.Source: §I.3 · p. 30
• MIRIX with GPT-4.1-mini costs $0.016 and scores 75.0 on MH-Doc QA, versus $0.043 and 66.0 for GPT-4.1-mini Long-Context.The analysis presents MIRIX as a middle ground for memory-intensive tasks.Source: §I.3 · pp. 30–31

Experimental setup

• The compute-matched ablation tests Banking77 for TTL and Book Summarization for LRU using GPT-4.1-mini.The three budget levels are low (4K tokens), medium (40K), and high (100K+); RAG and MIRIX use Top-K limits to match LC token budgets.Source: §J.1 · p. 31
• Book Summarization uses a random 30-book subset, and compute load is compared by total processed tokens rather than equalizing forward passes.The authors say forcing agentic models to one forward pass would strip away core reasoning capabilities.Source: §J.1 · p. 31

TTL: The Efficiency-Capacity Trade-off

• At 4K, RAG (83.0) beats LC (74.0); at 40K they equalize (90.0), while at high budget LC reaches 93.0 versus RAG 88.0.The authors attribute RAG’s high-budget decline to retrieved noise.Source: Table 18 · p. 31, §J.1 · p. 31

LRU: The Information Threshold Effect

• At low and medium budgets, all architectures score below 20.0; with full-text access at high budget, all achieve 39.0 and RAG nearly matches LC.The authors conclude that meeting the full-information threshold, rather than architecture, determines success on this task.Source: Table 18 · p. 31, §J.1 · p. 31

K.1 Prompt Design Specification

• The standardized dialogue prompt wraps chunks in simulated User-Assistant turns, gives each a memorization instruction, and includes task-specific instructions.The same template is used across long-context, RAG, and agentic systems to avoid prompt inconsistency in performance comparisons.Source: §K.1 · p. 32
• For selective forgetting, prompts index facts by serial number and instruct agents to prefer the newest fact when resolving conflicts.The appendix also tests an always-prefer-later policy and a conservative policy requiring explicit negation before overwriting.Source: §K.1 · p. 32, §K.2 · p. 32

K.2.1 Ablation Results

• The overwrite-policy ablation results are reported in Table 19 for selective-forgetting tasks measured by accuracy.The experiment compares the two overwrite policies on GPT-4.1-mini.Source: §K.2 · p. 32, §K.2.1 · p. 32, Table 19 · p. 32

K.2.2 Key Insights

• Policy A slightly raises FC-SH from 36.0 to 40.0 but drops FC-MH to 4.0, so its single-hop improvement does not generalize to complex multi-hop reasoning.The authors say prompting can help simple fact retrieval but cannot effectively propagate updates through multi-step reasoning chains.Source: §K.2.2 · p. 32
• Policy B lowers average performance by 4.5 points; its explicit-negation rule increases cognitive load and prevents retrieval of valid updates.The authors conclude selective forgetting requires dedicated memory mechanism design rather than prompt engineering alone.Source: §K.2.2 · pp. 32–33

L.1 Validity of LLM-as-a-Judge Evaluation

• On LongMemEval, Wu et al. (2025) report 98.0% agreement between a GPT-4o judge and human annotations; Yen et al. (2024) find GPT-4o judgments mostly align with human evaluations for summarization.The authors conclude their LLM-as-a-judge setup reliably reflects human evaluation for these benchmark tasks.Source: §L.1 · p. 33

L.2 Rationale for Chunked Input Format

• Chunked inputs approximate streaming interactions by accumulating real-world inputs into discrete pieces for inference, matching memory agents’ incremental processing.The paper contrasts this sequential format with standard long-context evaluations that use a static full document.Source: §L.2 · p. 33