PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

Yuanzhe Hu, Yu Wang, Julian McAuley

arXiv:2507.05257v4cs.CLcs.AI

TL;DR

Memory quality in LLM agents lacks a unified benchmark that reflects incremental, multi-turn information processing and covers all four core competencies. MemoryAgentBench reconstructs long-context datasets, adds EventQA and FactConsolidation, and evaluates diverse memory-agent strategies; existing methods are effective on some tasks but still face significant challenges in others.

  • Problem

    LLM-agent evaluations largely focus on reasoning and execution, while memory agents’ long-term memorization, updating, and retrieval remain under-evaluated by incomplete or static benchmarks.

  • Method

    MemoryAgentBench converts existing datasets into incremental multi-turn interactions, adds EventQA and FactConsolidation, and evaluates long-context, RAG, and commercial memory agents.

  • Results

    Existing memory agents are effective in some tasks but still face significant challenges in certain aspects of memory.

  • Takeaways & Limitations

    MemoryAgentBench provides a unified testbed for assessing how agents store, update, and use long-term information across four memory competencies.

  • Takeaways & Limitations

    LongMemEval remains constrained by limited topical diversity and less realistic interaction patterns.

Abstract

from arXiv · show

Recent benchmarks for Large Language Model (LLM) agents primarily focus on evaluating reasoning, planning, and execution capabilities, while another critical component-memory, encompassing how agents memorize, update, and retrieve long-term information-is under-evaluated due to the lack of benchmarks. We term agents with memory mechanisms as memory agents. In this paper, based on classic theories from memory science and cognitive science, we identify four core competencies essential for memory agents: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Existing benchmarks either rely on limited context lengths or are tailored for static, long-context settings like book-based QA, which do not reflect the interactive, multi-turn nature of memory agents that incrementally accumulate information. Moreover, no existing benchmarks cover all four competencies. We introduce MemoryAgentBench, a new benchmark specifically designed for memory agents. Our benchmark transforms existing long-context datasets and incorporates newly constructed datasets into a multi-turn format, effectively simulating the incremental information processing characteristic of memory agents. By carefully selecting and curating datasets, our benchmark provides comprehensive coverage of the four core memory competencies outlined above, thereby offering a systematic and challenging testbed for assessing memory quality. We evaluate a diverse set of memory agents, ranging from simple context-based and retrieval-augmented generation (RAG) systems to advanced agents with external memory modules and tool integration. Empirical results reveal that current methods fall short of mastering all four competencies, underscoring the need for further research into comprehensive memory mechanisms for LLM agents.

1 Introduction

Memory in LLM agents remains under-evaluated despite advances in reasoning and tool use. MemoryAgentBench addresses this gap with a multi-turn benchmark covering four complementary competencies and finds that existing agents struggle to master them all.

  • Current agent evaluations emphasize reasoning, planning, tool use, and code synthesis while largely under-examining memorization, storage, updating, and retrieval.
  • Memory agents use mechanisms such as parameters, vectors, textual histories, or external databases, with this paper focusing primarily on textual histories and external databases.
  • The benchmark evaluates four competencies: accurate retrieval, test-time learning, long-range understanding, and selective forgetting.
  • Existing benchmarks use either relatively short contexts or static long-context inputs, whereas memory agents incrementally absorb, abstract, consolidate, and update information over time.
  • MemoryAgentBench reconstructs datasets into multi-turn interactions, adds EventQA and FactConsolidation, and evaluates long-context, RAG, and commercial memory agents.
  • Existing memory agents perform effectively on some tasks but still face significant challenges in other aspects of memory quality.

2 Related Work

Prior work spans long-context, RAG, and memory-agent benchmarks, but coverage remains fragmented. Existing memory benchmarks are limited by short conversations, synthetic interactions, or incomplete evaluation across memory capabilities and agent categories.

  • Benchmarks on Long-Context and Memory: Long-context benchmarks progressed from approximately 20k–24k-token inputs to datasets reaching roughly 100k–200k tokens.
  • Benchmarks on Retrieval-Augmented Generation: RAG benchmarks evaluate knowledge-intensive tasks including open-domain question answering, fact checking, document ranking, and end-to-end retrieval systems.
  • Benchmarks on Long-Context and Memory: MemoryAgentBench compares datasets by capability coverage and evaluates coverage across Long-Context Agents, RAG Agents, and Agentic Memory.
  • Benchmarks on Long-Context and Memory: Memory-agent benchmarks include LOCOMO, LongMemEval, RealTalk, and StoryBench, but LOCOMO has relatively short conversations and LongMemEval uses synthetic conversations with limited topical diversity.
  • Benchmarks on Long-Context and Memory: The dataset overview organizes evaluations around important long-context capabilities and reports average context length using the GPT-4o-mini tokenizer.

3 MemoryAgentBench

MemoryAgentBench standardizes incremental, multi-turn evaluation across four memory competencies. It combines reconstructed long-context datasets with new tasks and compares long-context, RAG, and agentic-memory strategies under standardized interaction protocols.

  • Dataset Preparation: The benchmark reconstructs existing datasets and builds new datasets to evaluate each of the four memory competencies.
  • Accurate Retrieval: Accurate Retrieval uses document QA, LongMemEval (S*), and EventQA to test extraction, multi-turn recall, and temporal reasoning.
  • Test-Time Learning: Test-Time Learning evaluates classification and recommendation tasks that require applying behaviors or labels learned from previously seen examples.
  • Long-Range Understanding: Long-Range Understanding uses novel summarization and Detective QA tasks requiring global analysis of extended narratives.
  • Selective Forgetting: FactConsolidation orders original and contradictory facts so agents must resolve outdated knowledge against newly acquired information.
  • Datasets and Agents Formulation: Agents process chunks sequentially, incrementally update memory, and answer questions afterward using standardized prompts across evaluation categories.

4 Experiments

Experiments compare long-context, RAG, and agentic memory agents across four competencies and analyze chunk size, retrieval top-k, backbone models, and dataset validity. Results show complementary strengths: RAG favors accurate retrieval, long-context models favor test-time learning and long-range understanding, and selective forgetting remains difficult.

  • Experimental Setup: The evaluation covers Long-Context Agents, RAG Agents, and Agentic Memory Agents across four memory competencies.RAG agents include Simple, Embedding-based, and Structure-Augmented variants.
  • Overall Performance Comparison: RAG agents generally outperform GPT-4o-mini on Accurate Retrieval tasks by extracting crucial snippets from prior context.The comparison uses GPT-4o-mini as the reference backbone when no model is specified.
  • Overall Performance Comparison: Long-context models achieve the best performance on Test-Time Learning and Long-Range Understanding, while RAG and commercial memory agents retrieve only partial past information.The authors characterize this as a limitation for holistic understanding and learning across accumulated context.
  • Overall Performance Comparison: All existing methods struggle with Selective Forgetting, especially multi-hop situations involving outdated memories.The passage identifies forgetting out-of-date memory as a significant challenge for memory agents.
  • Ablation Study on Input Chunk Size: Smaller chunks can improve Accurate Retrieval when retrieval calls increase, but varying chunk size hurts Long-Range Understanding.Finer segmentation particularly improves relevance for embedding-based methods, whereas coherent-context integration suffers.
  • Ablation Study on Retrieval TopK: Increasing retrieval top-k generally improves performance across most tasks, but 10 chunks of 4096 tokens already produce approximately 40k input tokens.Settings with 20 retrieved chunks were not evaluated because of the resulting token volume and model-capacity demands.
  • Ablation Study on Backbone Model: A stronger backbone yields marginal gains for RAG agents but substantial gains for MIRIX, suggesting backbone advances may benefit Agentic Memory methods more.The comparison uses three backbone models and four representative memory agents.
  • Validation of Dataset FactConsolidation: FactConsolidation is solvable at short context lengths but becomes substantially harder at 32K, including a drop from 80.0 to 14.0 for o4-mini on FactCon-MH.This validation indicates current memory agents lack strong long-range reasoning for longer historical inputs.

5 Conclusion

The paper concludes that MemoryAgentBench provides a unified evaluation of memory agents across four competencies using multi-turn interactions. Its evaluation finds substantial limitations in current agents, while experiments on FactConsolidation show severe degradation as context length increases.

  • Conclusion: MemoryAgentBench evaluates accurate retrieval, test-time learning, long-range understanding, and selective forgetting in memory agents.The benchmark restructures existing datasets and adds EventQA and FactConsolidation.
  • Conclusion: The benchmark assesses how agents store, update, and use long-term information across multi-turn interactions.It evaluates long-context models, RAG systems, and commercial memory agents under a consistent protocol.
  • Conclusion: Current memory agents still show substantial limitations on dynamic memory updates and long-range consistency.This conclusion follows the paper’s evaluation across the benchmark’s four competencies.

Ethics Statement

The paper describes ethics safeguards for its dialog-based memory evaluation, including compliant data, privacy protections, safety-screened prompts, and usage guidance.

  • Ethics Statement: The study uses license-compliant corpora and reports collecting no personally identifiable information or data from minors.
  • Ethics Statement: The authors release safety-screened prompts and usage notes discouraging surveillance-oriented applications to reduce dual-use risks.
  • Ethics Statement: The work follows the ICLR Code of Ethics and associated author guidance, with potential impacts and mitigations documented.

Reproducibility Statement

The paper describes plans to release code, data, prompts, regeneration scripts, and execution environments to support reproducible evaluation. It also documents dataset curation and Accurate Retrieval as an operational benchmark competency.

  • Reproducibility Statement: The planned repository includes evaluation scripts, configurations, exact prompts, dataset releases, generation scripts with seeds, and end-to-end run recipes.
  • Reproducibility Statement: Pinned dependencies, a containerized environment, and hardware, CUDA/cuDNN, and operating-system details are intended to support deterministic reruns.
  • Reproducibility Statement: The paper documents dataset curation, metrics, average context length, and descriptions for the four competency categories in Table 2.
  • Reproducibility Statement: Accurate Retrieval is defined as identifying and retrieving important information dispersed throughout a long dialogue history.The task adapts long-context retrieval to agentic multi-turn conversations.

B.1.2 Details on AR datasets

The benchmark evaluates accurate retrieval, test-time learning, long-range understanding, and selective forgetting using multiple dataset formats and memory-agent implementations. Its AR datasets include long-document QA, dialogue QA, and event-continuation tasks, while TTL, LRU, and SF assess complementary memory abilities.

  • Accurate Retrieval datasets: Document QA constructs 197K- or 421K-token documents containing gold passages and scores answers with substring exact match.The datasets combine shuffled, deduplicated short documents while preserving the passage needed to answer each question.
  • Accurate Retrieval datasets: LongMemEval forms five chronological dialogue histories of approximately 355K tokens and evaluates open-ended answers with a GPT-4o judge.The reported metric is the proportion of satisfactory responses.
  • Accurate Retrieval datasets: EventQA uses five books, extracts 101 character events, and tests temporal continuation through six-way multiple-choice questions.Agents receive up to five previous events; results average 100 questions per book and then across all five books.
  • Benchmark competencies: Four task families evaluate accurate retrieval, test-time learning, long-range understanding, and selective forgetting as complementary memory-agent competencies.These categories cover retrieval fidelity, incremental skill acquisition, holistic comprehension, and updating outdated information.
  • Test-Time Learning datasets: TTL evaluates incremental task learning through multi-class classification and movie recommendation using long contexts of labeled examples or dialogues.MCC uses average accuracy, whereas recommendation uses Recall@5 over 20 recommended movies.
  • Selective Forgetting: Selective forgetting tests whether agents resolve contradictions by discarding outdated information while preserving relevant evidence when retrieval requires it.The SF setup orders counterfactual edit-pair sentences so newer information carries the answer.

E.4 Results on Different Context Length Ablation Study

The context-length ablation compares long-context and memory-based agents across increasing input lengths, while additional analyses examine chunk size, latency, and resource settings. Long-context performance declines as contexts grow, whereas Mem0 and Cognee remain below their backbone even at shorter lengths.

  • Context-length performance: Mem0 and Cognee perform significantly below their GPT-4o-mini backbone even when the context length is relatively small.This comparison concerns RAG-based agents under the reported scaling experiment.
  • Efficiency analysis: Smaller chunk sizes substantially increase memory-construction time, especially for HippoRAG-v2, Mem0, Cognee, and MemGPT.The latency analysis reports separate Memory Construction and Query Execution measurements.
  • Experimental settings: Experiments use task-dependent chunk sizes, with 512 for synthetic AR and SF contexts and 4096 for several continuous-text, TTL, and costly memory-agent settings.Embedding-model GPU memory usage is measured separately on a single A100 80GB GPU.

G Task Rationale and Justification for Selective Forgetting Task

The selective-forgetting task is motivated as a core memory-management capability for handling finite storage and interference between outdated and updated information. Its controlled synthetic design distinguishes proactive removal from ordinary knowledge updating while enabling reproducible evaluation.

  • Theoretical Necessity: Selective forgetting addresses the universal need to discard outdated, redundant, or superseded information when memory capacity is finite.The task treats selective forgetting as a controlled proxy for maintaining concise representations without conflicting signals.
  • Distinction from Previous Settings: Unlike knowledge updating, the task emphasizes proactive removal of non-essential information to free cognitive and contextual space.This frames selective forgetting as distinct from merely overwriting old facts with new conflicting ones.
  • Controlled Synthetic Setting: The synthetic setting isolates selective forgetting from confounding factors because naturalistic histories over 100K tokens lack unambiguous ground-truth annotations for what should be forgotten.The authors present controlled construction as supporting reproducible, apples-to-apples comparisons across architectures.
  • TTL terminology: TTL is defined as acquiring task-specific skills and rules incrementally from interaction history and applying them to unseen inputs at inference time.The design separates acquisition from evaluation on held-out inputs.
  • TTL terminology: TTL differs from retrieval because it measures learning from labeled examples rather than recalling static facts from interaction history.The benchmark uses tasks such as multi-class classification and personalized movie recommendation.

H.2 Zero-Shot Baseline Validation Experiments

The zero-shot experiments test whether TTL performance depends on historical examples rather than pretrained knowledge. Models are near chance without the history, while GPT-4o-mini performs substantially better with full memory.

  • Experimental design: Zero-shot evaluation removes the historical example sequence to test TTL without an opportunity for test-time learning.The experiment compares mainstream LLMs without history against GPT-4o-mini with full memory.
  • Results: Below 4% average accuracy is observed across both TTL tasks in the zero-shot setting.The tasks are Multi-Class Classification on Banking77 and personalized Movie Recommendation.
  • Results: 48.6% average accuracy is achieved by GPT-4o-mini with the full historical example sequence, a 45.2 percentage point absolute improvement.The reported comparison supports the benchmark’s intended dependence on interaction-history learning.
  • Cost analysis: The cost analysis compares Long-Context models, RAG agents, and MIRIX under context caching and Top-K=10 RAG settings.Costs use OpenAI pricing specified as of November 2025, while embedding indexing is excluded as a one-time expense.

I.2 Cost-Performance Results

The benchmark compares amortized inference cost per query with performance across four datasets differing in context length and reasoning complexity. RAG can be substantially cheaper than long-context models, while agentic memory can improve both cost and performance on memory-intensive tasks.

  • Evaluation Measure: Table 17 reports amortized inference cost per query and Accuracy/Score across four representative datasets.Costs are amortized over question sets sharing the same context.
  • RAG Efficiency: BM25 costs < $0.001–$0.006 per query but scores 52.1 versus 63.4 for long-context models on Detective QA.The cost advantage does not extend to tasks requiring global reasoning.
  • Long-Context Scaling: Upgrading from GPT-4o-mini to GPT-4.1-mini increases MH-Doc QA cost from $0.010 to $0.043 per query.The comparison highlights steep cost growth with stronger long-context backbones, even with caching.
  • Agentic Memory: MIRIX with GPT-4.1-mini costs $0.016 and scores 75.0 on MH-Doc QA, versus $0.043 and 66.0 for raw GPT-4.1-mini long-context inference.This result places agentic memory between retrieval and full-context scaling in the cost-performance comparison.

J.1 Experimental Setup

The experiments use compute-matched budgets, incremental dialogue chunks, standardized prompts, and overwrite-policy ablations to evaluate memory-agent competencies. Results show budget-dependent trade-offs for test-time learning, an information threshold for long-range understanding, and limited success from prompting alone on selective forgetting.

  • Experimental Design: The compute-matched study evaluates TTL and LRU with GPT-4.1-mini at Low, Medium, and High token budgets.Low, Medium, and High correspond to 4K, 40K, and 100K+ tokens, respectively.
  • Experimental Design: For Book Summarization, a random 30-book subset is evaluated using total processed tokens rather than equalized forward passes.The setup preserves agentic models’ multi-step reasoning capabilities.
  • Test-Time Learning: At 4K tokens, RAG scores 83.0 versus 74.0 for long-context models on TTL; at 100K+, long-context models reach 93.0 while RAG reaches 88.0.At 40K, performance equalizes at 90.0; RAG’s high-budget decline is attributed to retrieved noise.
  • Long-Range Understanding: On LRU, all architectures score below 20.0 at Low and Medium budgets, while meaningful performance appears only at the High budget, around 39.0.The reported pattern indicates that full-book summarization requires a full-information threshold.
  • Experimental Design: Inputs are delivered as simulated User-Assistant dialogue chunks with memorization instructions to trigger incremental memory processing.Selective Forgetting prompts index facts by serial number and require newer facts to override conflicts.
  • Validity Controls: The benchmark uses one standardized prompt template across long-context, RAG, and agentic systems to attribute observed gaps to memory mechanisms rather than prompt inconsistency.The authors specifically cite RAG failure on multi-hop Selective Forgetting as an example.
  • Selective Forgetting: Policy A raises FC-SH from 36.0 to 40.0 but lowers FC-MH to 4.0, while Policy B reduces average performance by 4.5 points.These ablations test aggressive later-fact preference versus conservative overwrite only after explicit negation.
  • Evaluation Validity: LLM-as-a-judge evaluation is acknowledged as potentially non-optimal, although GPT-4o achieves 98.0% agreement with human annotations on LongMemEval.The setup follows prior work and reports alignment with human evaluation for the benchmark’s tasks.
Loading 2507.05257v4…