Source-linked AI summary
MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks
Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu, Tzu-Ping Chen, Lang Yin, Ze Chen, Tong Arthur Wu, Siru Ouyang, Zihan Wang, Jiaxin Pei, Julian McAuley, Yejin Choi, Alex Pentland
TL;DR
Existing evaluations largely separate memory recall from agent action, although realistic tasks require memory to guide decisions across interdependent sessions. MEMORYARENA introduces a unified multi-session benchmark with feedback-driven memory use across four task environments and finds low success among state-of-the-art memory-augmented agents. The results expose persistent challenges in maintaining and reusing task-relevant state across sessions.
Problem
Existing benchmarks assess memory recall or single-session action separately, leaving memory-guided decision-making across interdependent sessions insufficiently evaluated.
Method
MEMORYARENA evaluates agents in human-crafted, multi-session tasks where memory is updated from environment interactions and conditions later actions.
Results
State-of-the-art memory methods achieve low success rates in MEMORYARENA, with agents often making subtask progress but failing to integrate it into globally consistent solutions.
Takeaways & Limitations
Effective agent memory requires maintaining and reusing task-relevant information across interdependent sessions, not merely recalling previously seen facts.
Takeaways & Limitations
Current memory systems and agents have limited capacity to preserve, update, query, and integrate task-relevant state for belief tracking.
Abstract
from arXiv · showhide
Existing evaluations of agents with memory typically assess memorization and action in isolation. One class of benchmarks evaluates memorization by testing recall of past conversations or text but fails to capture how memory is used to guide future decisions. Another class focuses on agents acting in single-session tasks without the need for long-term memory. However, in realistic settings, memorization and action are tightly coupled: agents acquire memory while interacting with the environment, and subsequently rely on that memory to solve future tasks. To capture this setting, we introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops. The benchmark consists of human-crafted agentic tasks with explicitly interdependent subtasks, where agents must learn from earlier actions and feedback by distilling experiences into memory, and subsequently use that memory to guide later actions to solve the overall task. MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning, and reveals that agents with near-saturated performance on existing long-context memory benchmarks like LoCoMo perform poorly in our agentic setting, exposing a gap in current evaluations for agents with memory.
1. Introduction
Existing memory benchmarks test recall or single-session action separately, but realistic agentic tasks require memory and action to operate together across dependent sessions. MEMORYARENA introduces this evaluation setting through long-horizon, interdependent tasks in a Memory-Agent-Environment loop.
- Recall-focused benchmarks assess memorization from static conversations or text but do not require agents to use memory for future actions.
- Single-session agent benchmarks evaluate action in dynamic environments without requiring persistent memory across episodes.
- MEMORYARENA treats memorization and action as inseparable through feedback-driven memory updates that condition later action selection across sessions.
- MEMORYARENA introduces human-crafted tasks with interdependent subtasks, where later actions depend on information acquired earlier.
- The benchmark spans bundled web shopping, preference-constrained group travel, progressive information search, and sequential formal reasoning, with tasks averaging 57 action steps and over 40k-token traces.
- State-of-the-art agents with long-context, RAG, or external memory systems achieve low task completion in MEMORYARENA despite strong performance on existing memory benchmarks.
2. Related Works
Prior work evaluates memory mainly through recall and agent ability mainly through interactive execution, leaving persistent memory use in interdependent task sequences insufficiently tested. MEMORYARENA distinguishes this setting by requiring cross-task causal dependence and end-to-end sequential completion.
- Long-context and conversational benchmarks test whether models retain and recall information from extended inputs or prior interactions.
- Interactive benchmarks evaluate reasoning, action selection, and tool use in dynamic environments, but typically use single-session independent tasks without persistent cross-episode memory.
- Several newer benchmarks retrieve facts or parameters from static tool-call traces for post-hoc queries rather than testing memory’s influence on future execution.
- MEMORYARENA enforces cross-task causal dependence and evaluates whether agents distill experiences into reusable knowledge that informs later decisions.
3. MEMORYARENA: Agent Memory in Memory-Action-Environment Loops
MEMORYARENA combines four long-horizon environments with sequential subtasks whose later decisions depend on earlier interactions. Its Memory-Agent-Environment loop retrieves task-relevant memory during action and updates persistent memory after each subtask.
- Web Navigation: Bundled Web Shopping: Bundled Web Shopping tests compatibility and preference consistency across purchases made over time.
- Compositional Information Seeking: Progressive Web Search: Progressive Web Search adds conditions across search steps, requiring the final answer to satisfy all previously introduced constraints.
- Preference-constrained Planning: Group Travel: Group Travel Planning adds travelers incrementally, requiring recall of prior activities and preferences across interdependent itinerary constraints.
- Sequential Formal Reasoning: Math & Physics: Formal Mathematical Reasoning and physics tasks use expert-curated chains of intermediate statements extracted from research-paper derivations.
- Memory-Agent-Environment Loop: Unlike single-session interaction, later subtasks depend on earlier information and therefore require persistent state across sessions.
- Memory-Agent-Environment Loop: The memory system retrieves relevant information during each action step and updates its persistent store after subtask completion.
4. Experiments
MemoryArena evaluates long-context, external-memory, and RAG agents on challenging multi-session tasks requiring interdependent state tracking. Results show broadly low completion and progress, depth-related performance decay, latency tradeoffs, and setting-dependent benefits from memory mechanisms.
- Evaluation Metrics and Main Results: Task Success Rate measures fully solved tasks, while Progress Score captures progress across subtasks and exposes partial success that does not yield globally consistent solutions.Success requires satisfying all group members in shopping and travel tasks, or correctly completing the final subtask in search and formal reasoning.
- Main Results: All methods exhibit low Success Rate and Progress Score, with two environments near zero and Group Travel Planning near zero across methods.Travel planning requires a 30-slot itinerary with preferences and interdependent constraints, creating simultaneous memorization and long-chain reasoning demands.
- Results on Interdependent Subtasks: No method maintains a flat success-rate region as subtask depth increases, indicating that current memory and retrieval mechanisms do not reliably sustain long-horizon execution.Decay rates vary by setting: RAG is more robust than external memory when tasks require precise reuse of earlier intermediate results, activities, or time slots.
- Latency Evaluations and Main Results: External memory consistently increases latency, RAG falls between external memory and long-context agents, and memory-operation complexity alone does not predict execution time.External memory helps in Progressive Web Search and Formal Reasoning, where long traces or complex reasoning exceed effective long-context capacity; it does not consistently outperform long-context history overall.
- MEMORYARENA as a POMDP Testbed: Current memory and RAG systems still yield low Task Success Rate because preserving task-relevant belief state and integrating retrieved information remain complementary bottlenecks.The paper identifies limitations in both memory mechanisms’ state preservation and agents’ querying and interpretation of memory outputs.
5. Conclusions
MemoryArena evaluates agent memory as a functional part of multi-session tasks with interdependent subtasks, rather than as isolated recall. Its results reveal persistent difficulty maintaining and reusing memory across sessions.
- MemoryArena uses curated multi-session tasks with interdependent subtasks to assess whether memory supports agent decision-making.
- State-of-the-art agent memory methods achieve low success rates in MemoryArena.
- The benchmark exposes persistent challenges in maintaining and reusing memory across interdependent sessions.
A. Appendix: More data details
The appendix illustrates the bundled shopping task, where an agent must purchase technically compatible items within a budget and follow specified search and purchase procedures.
- The bundled shopping task asks an agent to purchase a technically compatible item bundle that fits the budget.
- The shopping instructions require comparing all candidates, keeping the combined cost at or below $220, and searching for products individually.
- Figure 4 presents a data example for the bundled web shopping task.
An Example Data from the Group Travel Planning
The appendix presents examples and construction details for MemoryArena tasks, including structured travel planning, progressive web search, and formal reasoning, alongside a rule-based shopping-data pipeline.
- The travel-planning agent creates plans using available tools over structured environment tables.
- The appendix identifies examples for Group Travel Planning, Progressive Web Search, and formal reasoning with iterative problem solving.
- The shopping-data pipeline analyzes product categories, filters subcategories with more than 150 items, and selects the top five root categories.
- A three-level screening template adds dependency and rejection maps to basic product attributes, enforcing compatibility and excluding conflicting combinations.
- All constructed results undergo manual human inspection.
STEP 3: DATA INSTANTIATION AND TASK CONSTRUCTION
Task construction instantiates rule-based item chains from WebShop data, adds compatible and incompatible distractors, injects preferences, and produces manually inspected test samples.
- The pipeline retrieves logically valid item-chain combinations from the WebShop dataset using constructed rule templates.The preliminary candidate set contains tens of thousands of valid combinations.
- Each chain level receives compatible distractors and incompatible hard negatives to test understanding of dependency and exclusion constraints.Two compatible and two incompatible distractors are selected for each level.
- The construction process randomly selects among highest average rating, highest price, and lowest price preferences to determine the ground truth.
- Extracted attributes, candidates, and instructions are encapsulated in a standardized prompt framework for the shopping agent.
- The final evaluation contains 150 high-quality test samples, all manually inspected by annotators.
B. Reproducible Experiment Setups
The experiments evaluate memory systems across four environments using structured prompts, unified interfaces, and environment-specific settings. Memory is retrieved at the beginning of each subtask to reduce retrieval cost while preserving shared-skill coverage.
- MemoryArena evaluates memory systems across bundled web shopping, group travel planning, progressive web search, and sequential formal reasoning.
- Memory is retrieved once at the beginning of each subtask to cover shared skills while reducing retrieval frequency and cost.
- Bundled Web Shopping: WebShop tasks require sequentially completing multiple purchase sub-goals while satisfying global compatibility and preference constraints.
- The evaluation uses unified action-constraint prompts, capped interaction rounds, context budgets, and a common memory-system interface.
- Progressive Web Search: Progressive web search decomposes complex questions into self-contained subqueries that can be answered independently.
- Sequential Formal Reasoning: Formal reasoning experiments use temperature 0 and an 8192-token output limit for reproducible symbolic reasoning traces.
C.1. More Latency Results
This section reports task-level latency and provides a table describing latency in memory systems.
- Task-level latency is reported for the evaluated environments.
- The latency results complement the benchmark’s effectiveness evaluation with a systems-cost measure.
- Table 5 presents latency in memory systems in seconds.
C.2. Case study: Performance Analysis on Different Models in MEMORYARENA
The case studies compare models and memory systems across bundled web shopping, progressive web search, and sequential formal reasoning. They illustrate satisficing, retrieval failures, lost context, semantic drift, and downstream constraint violations.
- Bundled Web Shopping: GPT-5.1-mini immediately purchases the first relevant result, whereas Gemini and Claude backtrack to select a better-fitting option.
- Bundled Web Shopping: BM25 retrieval misses the prior “Compact” attribute and violates the negative constraint, while Long Context selects the “Articulating” option.
- Group Travel Planning: Long-context systems can lose details from the beginning of group-travel histories despite MemGPT achieving the best memory precision.
- Group Travel Planning: Memory retrieval failure in group travel planning causes drift from the finalized seed plan and a downstream dinner constraint violation.
- Progressive Web Search: Progressive web search case studies compare model memory retrieval and different memory systems.
- Progressive Web Search: The progressive-search examples include a correct identification of Matthew Arnum Barnor, while another trace exhibits semantic drift by answering a different historical question.
- Sequential Formal Reasoning: Formal-reasoning case studies compare memory systems, including a bound on L(h_t, u_t) that holds with probability at least 1 −δ/4.