PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

Yuyao Wang, Zhongjian Zhang, Mo Chi, Kaichi Yu, Yuhan Li, Miao Peng, Bing Tong, Chen Zhang, Yan Zhou, Jia Li

arXiv:2605.18421v2cs.CLcs.AIcs.LG

TL;DR

Agent memory remains under-evaluated because existing benchmarks do not systematically assess how memory stores, updates, and reuses information over time. EvoMemBench provides a standardized evaluation across memory scope and content, comparing representative methods with strong long-context baselines. The results show that memory is not a general solution: its benefits vary with context, task difficulty, and whether stored experience matches the task.

  • Problem

    Existing benchmarks do not systematically evaluate agent memory across its evolving knowledge and execution demands within and across episodes.

    Source: §1 · pp. 1–2

  • Method

    EvoMemBench evaluates four settings formed by combining in-episode versus cross-episode scope with knowledge-oriented versus execution-oriented memory, using a standardized comparative protocol.

    Source: §1 · pp. 1–2

  • Results

    Memory methods do not perform uniformly: strong memory-free long-context baselines remain competitive, and memory helps most when context is insufficient, tasks are difficult, or stored experience matches the target decision process.

    Source: §5.2.5 · p. 9

  • Takeaways & Limitations

    EvoMemBench is intended to support research on more effective memory systems for LLM-based agents.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution. However, memory is also essential for agents, as it enables them to store, update, and retrieve information over time. This ability remains under-evaluated, largely because existing benchmarks do not provide a systematic way to assess memory mechanisms. In this paper, we study agent memory from a self-evolving perspective and introduce EvoMemBench, a unified benchmark organized along two axes: memory scope (in-episode vs. cross-episode) and memory content (knowledge-oriented vs. execution-oriented). We compare 15 representative memory methods with strong long-context baselines under a standardized protocol. Results show that current memory systems are still far from a general solution: long-context baselines remain highly competitive, memory helps most when the current context is insufficient or tasks are difficult, and no single memory form works consistently across all settings. Retrieval-based methods remain strong for knowledge-intensive settings, whereas procedural and long-term memory methods are more effective for execution-oriented tasks when their stored experience matches the task structure. We hope EvoMemBench facilitates future research on more effective memory systems for LLM-based agents. Our code is available at https://github.com/DSAIL-Memory/EvoMemBench.

1 Introduction

Existing benchmarks leave important aspects of agent memory unevaluated, while agents need memory that evolves within and across tasks. EvoMemBench organizes evaluation by memory scope and content, covering four settings and providing a standardized comparative protocol.

1 Introduction

  • Self-evolving memory must retain and revise knowledge and task-relevant execution state within episodes, then accumulate reusable knowledge and execution experience across episodes.The framework treats scope and content as complementary axes.Source: §1 · p. 1
  • Existing benchmarks do not jointly evaluate in-episode and cross-episode evolution while distinguishing knowledge-oriented and execution-oriented demands.Some closely related benchmarks are also not fully open-sourced, limiting reproducible comparison.Source: §1 · p. 2
  • EvoMemBench evaluates four settings formed by combining in-episode versus cross-episode scope with knowledge-oriented versus execution-oriented memory.These settings test whether memory systems can update and reuse information over time.Source: §1 · p. 2
  • The paper presents EvoMemBench as a unified, open benchmark and provides a standardized comparison of representative memory methods across its four forms.The comparative analysis examines when memory helps, hurts, and matches particular agent-memory demands.Source: §1 · p. 2

2 Related Work

Prior work includes retrieval, long-term, procedural, and evolving memory methods, alongside benchmarks for conversational, incremental, continual, and agentic settings. These methods and benchmarks cover partial aspects of memory and use differing tasks and protocols, leaving no unified evaluation across memory scope and content.

2 Related Work

  • Memory methods differ in what they store and how they use it, including retrieved entries, structured persistent information, reusable procedures, and evolving memory architectures.The cited approaches are generally evaluated on different tasks and protocols, making comparisons across self-evolving needs difficult.Source: §2 · pp. 2–3
  • Retrieval-based, long-term, procedural, and evolving-context methods represent distinct approaches to storing, managing, and reusing agent experience.The reviewed methods are evaluated under differing tasks and protocols.Source: §2 · pp. 2–3
  • Text-based benchmarks assess conversational memory, while newer benchmarks examine incremental inputs, continual learning, sequential task streams, or memory-guided action.The review concludes that existing benchmarks lack a unified view of memory evolution across knowledge and execution, within and across episodes.Source: §2 · p. 3

3 Problem Formulation

The formulation models an agent that maintains memory while interacting with an environment over task episodes. It distinguishes memory by whether it persists within or across episodes and by whether it supports knowledge or execution.

3.1 Agents with Memory Mechanisms

  • At each step, the agent receives an observation, has an interaction history, and receives feedback; in tool-use tasks, feedback is the returned result.The interaction history can grow beyond the LLM's finite context window.Source: §3.1 · p. 3
  • The agent maintains a memory state as a compact representation of useful information from past interactions.The memory state addresses the growth of interaction history beyond the context window.Source: §3.1 · p. 3
  • The agent selects actions using both the current interaction context and its memory state.The policy is denoted by π.Source: §3.1 · p. 3
  • After feedback, the agent updates its memory state through a memory update function.The update function is denoted by U.Source: §3.1 · p. 3

3.2 Scope of Memory Evolution: In-Episode and Cross-Episode

  • In-episode memory retains useful information from earlier steps of the same episode to support later decisions in that episode.It is characterized as a task-level compact state.Source: §3.2 · p. 3
  • Cross-episode memory reuses information from prior task attempts in later episodes while avoiding episode-specific details that are no longer relevant.It is characterized as an experience-level compact state.Source: §3.2 · p. 3

3.3 Content of Memory Evolution: Knowledge and Execution

  • Knowledge evolution retains useful facts, constraints, preferences, domain rules, or task patterns and revises information when later evidence makes it outdated or inaccurate.It supports later reasoning or answering.Source: §3.3 · pp. 3–4
  • Execution evolution maintains action-relevant state and reusable experience, including task progress, intermediate results, unresolved subgoals, and procedures.It helps track what has been done, what remains, and which actions may be useful next.Source: §3.3 · p. 4

4 Benchmark Construction

EvoMemBench constructs six datasets to evaluate memory across in-episode and cross-episode knowledge and execution settings. Dataset selection prioritizes tasks with room for explicit memory gains, clear memory dependencies and verifiable outcomes, and coverage across domains and task formats.

4 Benchmark Construction

  • Six datasets are selected or reconstructed to assess memory where it can improve strong LLM performance, with clear dependencies, verifiable outcomes, and varied domains and task formats.The benchmarks are INEP-KNOW, INEP-EXEC, CROSSEP-KNOW, CROSSEP-TOOL, CROSSEP-WEB, and CROSSEP-EMB.Source: §4 · p. 4

4.1 In-Episode Knowledge Evolution

  • INEP-KNOW adapts incremental multi-turn MemoryAgentBench subsets to test whether agents retain and revise information as it arrives within one episode.It uses the Accurate Retrieval and Selective Forgetting subsets.Source: §4.1 · p. 4

4.2 In-Episode Execution Evolution

  • INEP-EXEC reconstructs noisy multiturn tool-use tasks with implicit references and ordered sub-turns, emphasizing execution-state tracking over complex planning.The reconstructed samples are assisted by GPT-5-mini and manually verified.Source: §4.2 · p. 4

4.3 Cross-Episode Knowledge Evolution

  • 120 contexts and 884 samples form CROSSEP-KNOW, grouping CL-Bench tasks by shared context and sequentially updating memory for later episodes.Its knowledge forms include domain facts, rule systems, procedures, and empirical patterns; contexts are divided into Easy, Medium, and Hard by baseline-score tertiles.Source: §4.3 · p. 4

4.4 Cross-Episode Execution Evolution

  • Cross-episode execution evaluates reusable experience in domain-specific tool-use, web-search, and embodied tasks, using BFCL multiturn tasks, web-search datasets, and ALFWorld.The datasets test reuse of tool-use patterns and API constraints, search and verification experience, and action routines and environment-specific information, respectively.Source: §4.4 · p. 5

5 Experiments

Experiments compare memory methods with long-context baselines across knowledge and execution settings. Results show that memory benefits depend on context, task difficulty, and whether stored experience matches the task.

5.1 Experiment Setup

  • The protocol compares memory-free agents with 15 memory methods using accuracy or success rate and token usage, with memory reset or reuse determined by setting.For in-episode tasks memory updates online and is cleared after each episode; cross-episode execution also tests transfer from a source subset to a target subset.Source: §5.1 · pp. 5–6

5.2.1 In-Episode Knowledge Evolution

  • Gemini-3-Flash ranks best on retention and revision, while explicit memory methods help retention more than revision and struggle with conflicting evidence.BM25, Mem0, and A-MEM improve over a weaker no-memory baseline on retention, but fall sharply on multi-hop FactConsolidation; the INEP-KNOW table reports accuracy and ranks.Source: §5.2.1 · p. 6, Table 3 · p. 6

5.2.2 In-Episode Execution Evolution

  • Against DeepSeek-V3.2, the best memory method raises overall success rate by +14.5 points at 16K, +14.0 at 32K, +7.8 at 64K, and +8.5 at 128K.Procedural methods lead overall, but short-term compression sometimes falls below the no-memory baseline, consistent with the need to preserve fine-grained execution details.Source: §5.2.2 · p. 7, Table 4 · p. 7

5.2.3 Cross-Episode Knowledge Evolution

  • Cross-episode knowledge memory can hurt easy tasks yet help hard ones, and ACE leads medium and hard splits by emphasizing knowledge application.On the easy split, DeepSeek-V3.2 scores 52.1 versus Qwen3-Emb-4B at 50.0; on the hard split, ACE scores 13.0 versus DeepSeek-V3.2 at 0.0. ACE's rule-system application scores are 74.9, 24.0, and 7.8 across easy, medium, and hard splits.Source: §5.2.3 · pp. 7–8, Table 5 · p. 8

5.2.4 Cross-Episode Execution Evolution

  • The strongest memory family differs by execution domain: retrieval-augmented memory leads Tool Using, general long-term memory Web Search, and procedural long-term memory Embodied AI.Their average ranks are 5.43, 5.38, and 5.60, respectively.Source: §5.2.4 · pp. 8–9, Table 6 · p. 8
  • Cross-environment transfer is more stable and positive for tool use than for ALFWorld, where transfer is more often neutral or negative.The paper relates this contrast to greater similarity in tool-use decision steps and larger differences in embodied task requirements and execution conditions.Source: §5.2.4 · p. 9, Figure 3 · p. 9, Figure 4 · p. 9

5.2.5 Summary

  • Current methods do not provide a general solution: long-context baselines remain competitive, and memory helps selectively when context, task difficulty, or experience alignment favors it.Retrieval remains strong for knowledge-oriented demands, while procedural and long-term memory can fit execution-oriented tasks when their stored form matches reusable task structure.Source: §5.2.5 · p. 9

6 Conclusion

EvoMemBench evaluates how agent memory evolves within and across episodes, covering knowledge-oriented and execution-oriented memory. Experiments find that current methods remain far from a general solution.

6 Conclusion

  • EvoMemBench combines in-episode versus cross-episode scope with knowledge-oriented versus execution-oriented content to evaluate evolving agent memory.The authors hope the benchmark will support research on more effective memory systems for LLM-based agents.Source: §6 · p. 10

A.1 Details of Datasets

The datasets span context learning, multi-turn tool use, agent memory, professional workflows, web navigation, and embodied decision-making. Their tasks assess knowledge acquisition, memory use, or long-horizon execution through task-specific outcomes.

CL-Bench

  • CL-Bench [6] tests acquiring and applying novel contextual knowledge across 500 contexts and 1,899 tasks, with performance measured by task-solving accuracy.It covers four categories and 18 subcategories, including procedural execution and empirical discovery.Source: §A.1 · p. 14

BFCL-V3-MultiTurn

  • BFCL-V3-MultiTurn evaluates API selection, executable arguments, and consistency across multi-turn, multi-step tool interactions.Performance is measured by function-calling accuracy and execution success rate.Source: §A.1 · p. 14

MemoryAgentBench

  • MemoryAgentBench [16] evaluates retrieval, test-time learning, long-range understanding, and selective forgetting through incremental multi-turn interactions.It measures memory retrieval and downstream task-completion accuracy as interaction lengths increase.Source: §A.1 · p. 14

xBench

  • xBench [4] assesses realistic long-horizon professional workflows across planning, coding, document processing, analysis, and web interaction.Outcomes include task success rate and execution quality across professional domains.Source: §A.1 · p. 14

WebWalkerQA

  • WebWalkerQA [36] tests sequential navigation across real-world webpages to gather evidence for multi-hop questions.It evaluates both navigation success and final question-answering accuracy.Source: §A.1 · p. 14

ALFWorld

  • ALFWorld [31] evaluates long-horizon household tasks involving navigation, object manipulation, and sequential decisions in partially observable environments.Performance is measured by task success rate and goal-completion accuracy.Source: §A.1 · p. 14

A.2 Details of Metrics

The paper reports task-performance and efficiency metrics matched to each evaluation setting, and compares memory approaches spanning long-context models and several memory-system families. The baseline set includes methods that retrieve information, maintain short- or long-term memory, or reuse procedural experience.

A.2 Details of Metrics

  • Knowledge settings use answer accuracy, while execution settings use success rate, a strict measure of full task accomplishment.Answer accuracy is the proportion of instances answered correctly; success rate reflects completion of the final goal.Source: §A.2 · p. 15
  • Progress score captures checkpoint-level partial completion, while average steps measures interaction length; lower values indicate fewer execution steps.Progress score offers finer-grained information than success rate, which rewards only fully completed tasks.Source: §A.2 · p. 15
  • Token usage counts input and output tokens across calls made by both the task agent and memory module, quantifying computational overhead.The count includes all LLM inference calls for an evaluation instance.Source: §A.2 · p. 15

A.3 Details of Baselines

  • Retrieval baselines include lexical BM25, dense Qwen3-Emb-4B [42], and graph-based GraphRAG [9] for retrieving or organizing information.GraphRAG uses an entity-centric knowledge graph and hierarchical community summaries.Source: §A.3 · p. 16
  • Short-term memory baselines compress or overwrite interaction histories into compact memory states, as described for MemAgent [39] and MemoBrain [27].MemAgent uses a fixed-size buffer; MemoBrain targets dynamic conversational reasoning.Source: §A.3 · p. 16
  • General long-term baselines include Mem0 [5] and A-MEM [37], which maintain salient interaction information or interconnected semantic notes for persistent retrieval.Mem0 supports incremental ADD, UPDATE, DELETE, and NOOP operations; A-MEM links notes across interactions.Source: §A.3 · pp. 16–17
  • Procedural and meta-evolution baselines reuse or refine experience, including AWM [33], ACE [41], and MemEvolve [40].AWM induces reusable workflows, ACE stores action templates, and MemEvolve iteratively updates memory structures using task feedback.Source: §A.3 · p. 17

B Additional Details on Implementations

The construction prompt turns dependent multi-action turns into sequential subtasks while preserving the original task semantics and action order. It prefers implicit references to earlier-established information so that later turns rely on memory without becoming ambiguous.

B.1 Prompt for INEP-EXEC and CROSS-TOOL Construction

  • The construction goal is to make examples more explicit multi-session sequences of dependent subtasks while preserving their original semantics.Later turns should refer implicitly, where possible, to objects, results, or state established earlier.Source: §B.1 · p. 18
  • Split ground truths must preserve ordered action slices without adding, deleting, reordering, or changing actions.Every action appears in exactly one slice, and split queries must remain aligned with their ground truths.Source: §B.1 · pp. 18–19
  • A turn is split only when multiple ground-truth actions form a meaningful sequential dependency, such as lookup followed by use of the result.Single-action, parallel-action, and low-level implementation-only turns are not split.Source: §B.1 · p. 18
  • Later queries should use natural, implicit references to earlier-established information when clarity permits, rather than restating resolved entities or values.The prompt cautions against making a query ambiguous or underspecified and preserves necessary explicit constraints.Source: §B.1 · p. 19

B.2 Details of Experiments.

The experiments standardize memory methods through utilization and update interfaces, with schedules that differ by evaluation setting. Supplementary figures present cross-environment transfer results for two cross-episode execution benchmarks, grouped across methods.

B.2 Details of Experiments.

  • Each method uses a utilize interface to return task-relevant memory and an update interface to incorporate newly observed information.The returned memory becomes part of the agent prompt.Source: §B.2 · p. 21
  • In-episode memory is reset per episode and updated by information block for knowledge tasks or by interaction turn for execution tasks.Cross-episode memory updates after each episode and is reused within the same context or environment group.Source: §B.2 · p. 21
  • For cross-environment execution transfer, memory is built on a source environment and frozen during evaluation on the target environment.For INEP-EXEC, context budgets are 16K, 32K, 64K, and 128K; when exceeded, early history is truncated while retaining the current task input and memory.Source: §B.2 · p. 21

C Additional Experimental Results

  • The supplementary results section includes full cross-environment transfer visualizations for CROSSEP-EMB and CROSSEP-TOOL.The supplied passages identify these as additional results but do not state outcome comparisons.Source: C Additional Experimental Results · p. 22, §C.1 · p. 22

C.1.1 CROSSEP-EMB

  • Figures 5 and 6 show CROSSEP-EMB transfer results, divided between BM25, GraphRAG, Mem0, and MemOS and a second group of four methods.The second group is MemoryOS, AWM, AgentKB, and ACE; captions do not state which methods perform best.Source: Figure 5 · p. 22, Figure 6 · p. 22

C.1.2 CROSSEP-TOOL

  • Figure 7 reports CROSSEP-TOOL cross-environment transfer results for BM25, GraphRAG, Mem0, and MemOS, while Figure 8 reports such results for MemoryOS, AWM, AgentKB, and ACE.Source: Figure 7 · p. 22, Figure 8 · p. 23

C.2 Additional Efficiency Results for Cross-Episode Settings

The cross-episode efficiency results report token usage for knowledge evolution and average steps and token usage for execution settings, with tables covering knowledge, embodied, tool, and web tasks.

C.3 Additional Results for INEP-EXEC

INEP-EXEC results include progress scores and efficiency comparisons across context lengths; the efficiency tables report average steps and token usage at four settings.

  • Complementary INEP-EXEC results include progress score and average steps under different context budgets.Source: §C.3 · p. 26
  • Progress scores are compared across memory methods and context lengths for INEP-EXEC.Unless otherwise specified, memory agents use DeepSeek-V3.2 as the backbone.Source: Table 14 · p. 26
  • At 16K and 32K context, INEP-EXEC efficiency comparisons report average steps and token usage, with lower values better.Source: Table 15 · p. 27, Table 16 · p. 27, Table 17 · p. 28
  • At 64K and 128K context, INEP-EXEC efficiency comparisons report steps or average steps and token usage, with lower values better.Source: Table 19 · p. 29, Table 20 · p. 29, Table 21 · p. 30

Related papers

Loading 2605.18421v2…