Source-linked AI summary
MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents
Haoran Tan, Zeyu Zhang, Chen Ma, Xu Chen, Quanyu Dai, Zhenhua Dong
TL;DR
Existing evaluations of LLM-agent memory often cover limited memory levels and interaction scenarios and omit efficiency and capacity. MemBench constructs a dataset and benchmark spanning factual and reflective memory, participation and observation scenarios, and multiple metrics, with experiments comparing memory mechanisms and base models. The paper concludes that the framework provides a more comprehensive and scalable evaluation while noting that its assessment is limited to structured data.
Problem
Previous memory evaluations often focus on factual memory and participation scenarios, while overlooking reflective memory, observation scenarios, efficiency, and capacity.
Method
MemBench constructs a dataset with factual and reflective memory across participation and observation scenarios and evaluates agents using accuracy, recall, capacity, and temporal efficiency.
Results
Experiments compare seven common memory mechanisms and multiple base models across the benchmark’s memory evaluations.
Takeaways & Limitations
The dataset and time-aware framework provide a more comprehensive and scalable basis for evaluating LLM-agent memory mechanisms.
Takeaways & Limitations
The evaluation is limited to memory for structured data, although the framework can examine how agents structure memories during user-agent interactions.
Abstract
from arXiv · showhide
Recent works have highlighted the significance of memory mechanisms in LLM-based agents, which enable them to store observed information and adapt to dynamic environments. However, evaluating their memory capabilities still remains challenges. Previous evaluations are commonly limited by the diversity of memory levels and interactive scenarios. They also lack comprehensive metrics to reflect the memory capabilities from multiple aspects. To address these problems, in this paper, we construct a more comprehensive dataset and benchmark to evaluate the memory capability of LLM-based agents. Our dataset incorporates factual memory and reflective memory as different levels, and proposes participation and observation as various interactive scenarios. Based on our dataset, we present a benchmark, named MemBench, to evaluate the memory capability of LLM-based agents from multiple aspects, including their effectiveness, efficiency, and capacity. To benefit the research community, we release our dataset and project at https://github.com/import-myself/Membench.
1 Introduction
MemBench addresses gaps in evaluating LLM-agent memory by covering multiple memory levels and interaction scenarios, while assessing effectiveness, efficiency, and capacity. It introduces a dataset and benchmark spanning factual and reflective memory in participation and observation settings.
- LLM-agent memory modules store critical information and accumulate experiences, supporting dynamic tasks and continual adaptation.
- Previous evaluations often emphasize factual memory, participation scenarios, or indirect performance measures while neglecting reflective memory, observation scenarios, efficiency, and capacity.
- It includes participation scenarios, where agents interact with users, and observation scenarios, where agents record user-provided information as observers.
- The proposed dataset covers factual and reflective memory, enabling evaluation of information extraction, cross-session reasoning, knowledge updating, temporal reasoning, and reflective summarization.
- The benchmark evaluates memory using accuracy, recall, capacity, and temporal efficiency.
- The authors release the dataset and project and describe the work as a more comprehensive benchmark with multi-scenario and multi-level content.
2 Related Works
LLM-based agents use memory to store observed information and recall relevant experiences across interactive applications, but existing evaluations incompletely cover memory scenarios, levels, and metrics. The paper addresses these gaps with broader scenario and memory coverage and a more comprehensive comparison framework.
- LLM-based agents are applied in recommendation, finance, and personal-assistant settings, where memory supports storing observed information and recalling relevant experiences for inference.
- Existing memory datasets primarily use dialogue-based participation scenarios and focus on factual memory, often overlooking observation scenarios and reflective memory.
- Existing evaluations use subjective scoring or indirect task-performance comparisons, while their evaluation metrics are not comprehensive.
- The paper emphasizes reflective memory and observation scenarios, adopts methods better suited to the agent’s memory process, and provides more comprehensive metrics.
3 Dataset Construction
The dataset expands memory evaluation across participation and observation scenarios, factual and reflective memory levels, and multiple evaluation metrics. It constructs user-relation graphs, dialogue or message-based test data, and questions targeting extraction, updating, reasoning, summarization, recall, capacity, and efficiency.
- Memory Dataset Construction: The construction extends an existing observation framework with participation dialogues, knowledge updating, assistant-response extraction, and reflective-memory generation.Participation conversations select low-level preference attributes, generate evidence dialogues, and insert intervening discussion to form fluent multi-turn sessions.
- User Relation Graph Sampling: User-relation graphs combine profiles with associated individuals, events, places, and items, while sampled high-level attributes support reflective memory.The construction extends factual-attribute sampling with reflective-attribute sampling using user-item relationships and ratings from recommendation datasets.
- Multi-scenario Memory: The dataset covers participation, where agents interact with users, and observation, where agents passively record user messages.Participation data uses multi-turn user-agent sessions, while observation data uses message lists.
- Multi-level Memory: Factual-memory questions test information extraction, temporal conversion, knowledge updating, and single- or multi-session reasoning over user and entity attributes.Questions can require converting indirect references such as “next Monday” into exact dates and integrating multiple entities’ attributes.
- Multi-level Memory: Reflective memory extracts and summarizes high-level preferences from repeated low-level factual expressions, such as inferring taste from liked dishes.Multiple expressions reinforce the inferred memory content for evaluation.
- Multi-metric Evaluation: The benchmark evaluates accuracy, recall, capacity, and temporal efficiency using multiple-choice questions, evidence dialogues, memory-load thresholds, and processing-time costs.Capacity is associated with a sharp accuracy decline at a critical memory-content threshold, although retrieval effectiveness can also determine accuracy.
4 Benchmark
MemBench evaluates memory mechanisms across time-aware participation and observation scenarios, factual and reflective memory, and effectiveness, efficiency, and capacity dimensions. Results show substantial degradation under longer contexts, while model choice and mechanism design affect accuracy and runtime.
- Experimental Settings: The benchmark simulates time-aware interactions, where earlier content must be recalled through memory rather than remaining directly available.Noise sessions create difficult tests averaging over 100k tokens per test.
- Experimental Settings: It implements seven memory mechanisms with Qwen2.5-7B as the base agent model while leaving action modules and other components unchanged.The mechanisms include FullMemory, RetrievalMemory, RecentMemory, GenerativeAgent, MemoryBank, MemGPT, and Self-Controlled Memory.
- Evaluations on Factual Memory: FullMemory, RetrievalMemory, and RecentMemory perform best on factual-memory Sub-dataset 1, but FullMemory and RecentMemory decline on Sub-dataset 2.The decline is attributed to target messages falling outside the memory window, with RecentMemory affected more strongly; MemGPT reads more slowly and MemoryBank writes more slowly.
- Evaluations on Reflective Memory: GenerativeAgent, MemGPT, and MemoryBank perform well on reflective-memory Sub-dataset 1 but decline substantially on Sub-dataset 2, whereas RetrievalMemory retains good results.Limited context windows or forgetting mechanisms may cause important memories to be lost after longer interactions.
- Memory Capacity: Both MemGPT and Self-Controlled Memory show a sharp accuracy decline as memory tokens increase, indicating an upper limit on retained memory performance.Figure 5 visualizes accuracy as memory-token count increases.
- Comparison of Different Inference Models: Under the same context length, the base model significantly affects performance; GPT-4o-mini is usually strongest, while model-dependent runtime differences are generally modest.Meta-Llama-3.1-8B-Instruct has weaker factual but relatively good reflective memory, and GPT-4o-mini substantially increases GenerativeAgent’s time consumption.
5 Conclusion
The paper presents a scalable benchmark for evaluating agent memory more comprehensively than prior settings. It combines scenarios, memory levels, time-aware interactions, multiple metrics, and seven memory mechanisms.
- MemBench provides a more comprehensive and scalable dataset for evaluating LLM-based agents’ memory mechanisms.
- The dataset covers participation and observation scenarios as well as reflective and factual memory.
- The time-aware framework simulates daily user-agent interactions and evaluates accuracy, recall, capacity, and temporal efficiency.
- The benchmark evaluates seven common memory mechanisms.
Limitations
The evaluation is limited to structured data represented as profiles of users and related entities, leaving other forms of reflective memory for future study.
- The dataset represents users and relevant entities as a graph of profiles, so the evaluation focuses on structured memory.The paper suggests examining profile construction and attribute capture to study memory structuring.
- Reflective-memory areas such as users’ emotional memory remain underexplored.
Ethics Statement
The dataset uses publicly available authorized data and acknowledges risks from LLM-generated content, including unintended bias or harmful output.
- Publicly available data are used under their respective research licenses, while LLM-generated content may introduce unintended biases or harmful outputs.The authors encourage responsible dataset use to reduce ethical risks.
A.1 User Relation Graph Example
The dataset’s user relation graph represents users and connected entities through profiles, while examples pair factual and reflective-memory questions with dialogue-based tests.
- User relation graphs include profiles for users, related individuals, events, items, and places.
- Item and place examples encode relationships, entity types, names, reviews, and associated contextual details.
- Reflective-memory mappings connect high-level attributes such as tastes or genres to lower-level examples including foods, movies, and books.
- Participation examples test memory through single-hop, comparative, aggregative, post-processing, and recommendation questions across dialogue rounds.
- Knowledge updating examples revise the Policing Forum duration from four days to one day.
- Multi-session examples combine recommendations, personal facts, and follow-up questions, including recalled movies, books, dishes, and a niece’s company.
B Detail Data Statics
The dataset documentation details reflective-memory prompts and dialogue-generation procedures for participation and observation settings.
- Reflective-memory questions select a user’s taste from predefined categories based on dishes the user likes.
- Role dialogue prompts generate JSON-formatted user-assistant conversations grounded in supplied entity information without introducing additional facts.
- Event dialogue prompts similarly generate conversations about a specified event, beginning from the user’s attendance and using only supplied information.
- Message prompts rewrite user inputs into smooth colloquial declarative sentences while preserving the original information.
- Event-message prompts additionally avoid second-person phrasing and require first-person references such as I, me, or my.
D.1 Reflective Result
The documentation identifies a 10k reflective-memory dataset and reports that its detailed results are provided in Table 10.
- The 10k-Reflective memory dataset’s detailed results are presented in Tab 10.
D.2 Facutal Result
Detailed factual-memory results are provided for reflective and factual participation and observation datasets, with tables organizing multiple question types and mechanisms.
- Detailed results are provided for the 10k-Factual-Partipation and 10k-Factual-Observation memory datasets.
- Table 10 reports results across different mechanisms and types for the 10k-Reflective memory dataset.
- Table 11 organizes 10k-Factual-Participation results by single-hop, multi-hop, comparative, aggregative, post-processing, knowledge-update, and assistant-session types.
- Table 12 organizes 10k-Factual-Observation results by single-hop, multi-hop, comparative, aggregative, post-processing, and knowledge-update types.