Source-linked AI summary
Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory
Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, Zhankui He, Yuanchen Bei, Xuying Ning, Mengting Ai, Yunzhe Li, Jingrui He, Ed H. Chi, Chi Wang, Shuo Chen, Fernando Pereira, Wang-Cheng Kang, Derek Zhiyuan Cheng
TL;DR
LLM memory research has largely emphasized passive conversational recall rather than accumulating and reusing experience across evolving task streams. Evo-Memory benchmarks self-evolving memory through sequential tasks, compares memory modules, and introduces ExpRAG and ReMem; its results show substantial performance gains, while generality remains bounded by model, modality, and environment coverage.
Problem
Existing evaluations mostly test passive conversational recall and rarely assess how LLM agents accumulate and reuse experience across evolving task streams.
Method
Evo-Memory restructures datasets into sequential task streams, evaluates over ten memory modules across diverse tasks, and introduces ExpRAG and ReMem for experience reuse.
Results
Self-evolving memory methods provide consistent improvements across single-turn reasoning and QA benchmarks, with ReMem achieving 0.65 average exact match and 0.85/0.71 API accuracy under Gemini-2.5 Flash.
Takeaways & Limitations
Memory can substantially enhance performance, but reliable stability and procedural reuse remain unresolved challenges for continually improving LLM agents.
Takeaways & Limitations
The study focuses on selected strong LLMs and primarily textual, goal-oriented tasks, leaving open-weight, multilingual, multimodal, and richer real-world evaluations for future work.
Abstract
from arXiv · showhide
Statefulness is essential for large language model (LLM) agents to perform long-term planning and problem-solving. This makes memory a critical component, yet its management and evolution remain largely underexplored. Existing evaluations mostly focus on static conversational settings, where memory is passively retrieved from dialogue to answer queries, overlooking the dynamic ability to accumulate and reuse experience across evolving task streams. In real-world environments such as interactive problem assistants or embodied agents, LLMs are required to handle continuous task streams, yet often fail to learn from accumulated interactions, losing valuable contextual insights, a limitation that calls for test-time evolution, where LLMs retrieve, integrate, and update memory continuously during deployment. To bridge this gap, we introduce Evo-Memory, a comprehensive streaming benchmark and framework for evaluating self-evolving memory in LLM agents. Evo-Memory structures datasets into sequential task streams, requiring LLMs to search, adapt, and evolve memory after each interaction. We unify and implement over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets. To better benchmark experience reuse, we provide a baseline method, ExpRAG, for retrieving and utilizing prior experience, and further propose ReMem, an action-think-memory refine pipeline that tightly integrates reasoning, task actions, and memory updates to achieve continual improvement.
1. Introduction
The introduction identifies a gap in static LLM memory evaluations: agents often recall facts but do not reuse experience or evolve memory across task streams. Evo-Memory addresses this with a streaming benchmark, unified memory evaluation, and methods for experience retrieval and continual refinement.
- Motivation: Existing LLM memory systems mainly retrieve static dialogue context, assessing conversational recall rather than experience reuse across evolving tasks.Conversational recall retrieves prior facts, whereas experience reuse abstracts reasoning strategies for future tasks.
- Motivation: Without experience reuse, agents repeatedly solve similar problems and fail to adapt across sessions.
- Motivation: Prior benchmarks examine sequential learning, lifelong learning, or conversational consistency but rarely model memory structure, updates, reasoning reuse, or trajectory reuse together.
- Evo-Memory: Evo-Memory restructures datasets into sequential task streams spanning multi-turn goal-oriented and single-turn reasoning or problem-solving tasks.Models must retrieve, adapt, and evolve memory after each interaction to test test-time evolution.
- Evo-Memory: The framework implements over ten representative retrieval-based, workflow, and hierarchical memory modules for studying adaptation behavior.
- Methods: ExpRAG retrieves prior task experiences, while ReMem integrates reasoning, action, and memory updates through an action–think–memory refine pipeline.The introduction presents both as approaches for examining experience reuse and continual improvement.
2. Related Work
Related work progresses from test-time adaptation toward continual self-improvement and from passive LLM memory toward richer management and conversational evaluation. These lines of work motivate evaluating self-evolving memory during deployment.
- Test-time learning: Test-time learning extends test-time adaptation toward continuous self-improvement during deployment.
- Test-time learning: Recent agent studies operationalize continual improvement through reflection, planning, and self-evolution.
- LLM memory: Early LLM memory systems passively stored recent dialogues or retrieved facts to compensate for limited context windows.
- LLM memory: Later work introduced differentiable read–write controllers and evaluations in realistic conversational settings.
3. Evo-Memory: Evaluating Self-Evolving Memory in LLM Agents
Evo-Memory formulates memory-augmented agents as iterative systems that retrieve and synthesize memory, produce outputs, and update memory across sequential tasks. It instantiates this setting with ExpRAG for experience retrieval and ReMem, which integrates reasoning, action, and memory refinement.
- Problem Formulation: Evo-Memory unifies memory-augmented agents as iterative systems that retrieve memory, construct task-specific context, generate outputs, and update memory over input sequences.The formulation covers retrieval, contextual construction, and memory evolution as a common cycle.
- Dataset Preparation: The benchmark converts static datasets into streaming task sequences where earlier tasks provide information or strategies for later tasks.At each step, agents process an input, retrieve and synthesize memory, predict an output, and update memory.
- ExpRAG: ExpRAG retrieves k similar structured experiences, conditions on them as in-context examples, and appends each new experience to memory.It provides one-shot experience reuse but lacks iterative reasoning and adaptive refinement during inference.
- ReMem: ReMem integrates Think, Act, and Refine operations in one decision loop, enabling active evaluation, reorganization, and evolution of memory during problem solving.Think decomposes tasks, Act executes operations or produces responses, and Refine exploits useful experiences, prunes noise, and reorganizes memory.
- ReMem: ReMem expands ReAct-style agents with explicit memory reasoning, making memory an adaptive component that interacts with reasoning in real time.The framework supports continual adaptation by coupling reflection with memory evolution.
4. Experiments
Evo-Memory evaluates self-evolving memory under streaming multi-task conditions using diverse benchmarks, agents, and memory architectures. Results show consistent gains, especially for ReMem and experience-retrieval methods in multi-turn settings.
- Experimental setup: The experiments address performance, memory effectiveness, task efficiency, sequence difficulty, and feedback effects across Evo-Memory task streams.The benchmark compares methods under unified search–predict–evolve conditions.
- Experimental setup: The benchmark spans factual knowledge, reasoning, mathematics, programming, tool use, and multi-turn goal-oriented interaction.Datasets include MMLU-Pro, GPQA-Diamond, AIME-24/25, ToolBench, Alf World, BabyAI, ScienceWorld, and PDDL tasks.
- RQ1: Cross-benchmark results: ReMem achieves 0.65 average exact match and 0.85/0.71 API accuracy under Gemini-2.5 Flash in single-turn reasoning and QA benchmarks.Evolving-memory methods show consistent but moderate improvements in these settings, while ExpRAG outperforms several more complex designs.
- RQ1: Cross-benchmark results: ReMem and ExpRAG reach 0.92/0.96 on BabyAI and 0.95/0.62 on ScienceWorld across Gemini-2.5 and Claude backbones.Performance gains are larger in multi-turn settings, where continual refinement supports procedural knowledge accumulation and long-horizon experience reuse.
- RQ2: Memory improvement: ReMem’s improvement correlates with within-dataset task similarity, with Pearson r=0.717 on Gemini 2.5 Flash and r=0.563 on Claude 3.7 Sonnet.PDDL and Alf World show larger gains because recurring task structures support experience reuse.
- RQ2: Memory improvement: ReMem reduces Alf World’s average completion steps from 22.6 to 11.5, while ExpRAG and ExpRecent also remain competitive.Across four environments, evolving-memory methods require fewer steps; lower is better.
- RQ3–RQ4: Robustness and feedback: ReMem reaches up to 0.94/0.97 success and progress in Hard→Easy sequences and remains robust when failed experiences are stored.Baselines vary substantially under difficulty shifts and degrade with unfiltered failures, whereas refinement supports selective memory utilization.
5. Conclusion
The paper concludes that self-evolving memory is underexplored because prior evaluations emphasize static conversational recall rather than experience reuse across task streams. Evo-Memory provides a streaming evaluation platform, and its results support task-level utilization and continual memory refinement as promising directions.
- Conclusion: Evo-Memory transforms static datasets into streaming trajectories to evaluate how agents retrieve, adapt, and refine memory through interaction.The benchmark spans factual, reasoning, mathematical, programming, and goal-oriented tasks.
- Conclusion: The results indicate that memory can substantially improve performance but remains fragile in stability and procedural reuse.ExpRAG and ReMem are introduced to study experience retrieval and integrated reasoning, action, and memory updates.
- Conclusion: Its dataset suite jointly evaluates factual recall, domain expertise, mathematical reasoning, and procedural memory in interactive settings.This supports unified evaluation of static and evolving capabilities across academic and real-world task types.
- Conclusion: The framework uses a shared retriever and top-k memory retrieval budget to compare memory methods consistently.The default retrieval budget is k=4, with adaptive reasoning modules operating over the same retrieval pool.
- Conclusion: Evo-Memory measures answer accuracy, success rate, step efficiency, and sequence robustness as complementary dimensions of memory quality.These metrics cover correctness, goal completion, reasoning economy, and adaptation under changing task sequences.
A.4. Methods
The study benchmarks memory architectures and update strategies for test-time evolution, spanning pipeline, adaptive, procedural, and proposed evolving-memory approaches. It introduces ExpRAG and ReMem as experience-reuse methods that integrate retrieval, reasoning, action, and memory refinement.
- Methods: The benchmark compares memory and agent architectures across two LLM backbones while isolating memory architecture and update strategy.The study uses Gemini-2.5 and Claude-3.5/3.7; backbone capability is not the focus.
- Agent Pipelines without Procedural Memory: ReAct uses only immediate context, while Amem caches recent observations and reflections without dedicated search or update policies.Amem provides a minimal form of experience reuse between memory-free and adaptive-memory agents.
- Adaptive Agentic Memory Methods: Adaptive methods support dynamic retrieval, reflection, and structured read, write, and update operations for agent-level memory.The unified interface maps retrieval to search and memory changes to updates.
- Memory-Based Agents for Procedural Memory: Procedural-memory methods encode reusable how-to knowledge and test how workflow induction and update mechanisms affect stability and transfer.The evaluated Dynamic Cheatsheet variants are retrieval-based and curated.
- Proposed: Evolving Memory Framework: ExpRecent stores condensed recent trajectories, ExpRAG adds test-time evolution to retrieval-augmented reasoning, and ReMem iteratively refines memory embeddings.ReMem interleaves reasoning, acting, and memory refinement during deployment.
B.1. Additional Experiments
Additional experiments report consistent ReMem advantages across model families, multi-turn embodied tasks, and single-turn reasoning benchmarks. The analyses also examine cumulative adaptation and dataset-dependent memory pruning.
- Cross-setting results: ReMem consistently outperforms conventional baselines and adaptive retrieval methods across multi-turn and single-turn tasks and model backbones.The multi-turn tasks include Alf World, BabyAI, PDDL, and ScienceWorld; single-turn tasks include AIME-24/25, GPQA, MMLU-Pro, and ToolBench.
- Memory pruning: 36.8% pruning is reported for GPQA, compared with 17.5% and 10.8% for AIME settings, indicating benchmark-dependent memory selectivity.The passage relates higher pruning to broader domain coverage and lower pruning to more concentrated problem types.
- Multi-turn cumulative performance: ReMem adapts faster and retains performance more stably than History across four interactive multi-turn datasets over long task sequences.Figure 6 reports cumulative accuracy over task sequences rather than learning curves.
- Single-turn cumulative performance: ReMem surpasses History after the cold-start phase as tasks accumulate on GPQA, ToolBench, and MMLU-Pro (Engineer) for both evaluated model variants.The comparison covers Gemini-2.5-Flash-Lite and Claude-3.7-Sonnet.
C. Potential Risks
The paper identifies reliability, robustness, and security risks in LLM-driven memory management, while documenting benchmark, licensing, privacy, and deployment assumptions. Its figures and prompts also specify how memory and task interactions are operationalized.
- Potential Risks: LLM-generated judgments can produce unreliable memory updates that affect subsequent retrieval and decision making.The paper also identifies adversarial or misleading interactions as potential sources of memory poisoning.
- Potential Risks: Memory-based agents may be vulnerable to attacks or poisoning through corrupted experiences introduced by adversarial interactions.Reliability, robustness, and security remain stated directions for future work.
- Study assumptions: The study uses established public benchmarks and proprietary models through official APIs under their respective usage policies.The datasets and models are used for research evaluation and inference rather than parameter training.
- Study assumptions: The evaluation uses external memory modules without modifying pretrained model parameters, limiting test-time learning to controlled interaction and inference.The underlying model weights remain unchanged.
- Data considerations: The benchmark materials state that some datasets may inherit bias or problematic content from web-scale sources, consistent with known limitations of their original benchmarks.The study does not introduce new user-generated content or collect personal data.
F. Statistics for Data
The benchmark statistics cover diverse reasoning, embodied-interaction, navigation, science, and symbolic-planning tasks. These datasets vary in domain breadth, interaction structure, and planning demands.
- Dataset statistics: MMLU-Pro contains approximately 12K multiple-choice questions across 14 subject areas, including engineering, economics, philosophy, and natural sciences.The listed major domains include 969 Engineering, 844 Economics, and 499 Philosophy questions.
- Dataset statistics: GPQA-Diamond contains 198 expert-curated graduate-level multiple-choice questions.The passage introduces GPQA-Diamond as part of the benchmark statistics.
- Dataset statistics: Alf World, BabyAI, ScienceWorld, and PDDL contain 134, 112, 90, and 60 tasks respectively, spanning household interaction, navigation, science, and symbolic planning.The tasks require multi-step planning, object manipulation, long-horizon reasoning, or goal-directed state transitions.
G. Computational Experiments
The experiments evaluate Evo-Memory across multiple LLM backbones under fixed API budgets, using task-appropriate metrics averaged over instances and trajectories. Implementation relies on standard Python tooling, vendor SDKs, PyTorch, and Weights & Biases.
- Experiments compare Gemini-2.5 Flash, Flash-Lite, Pro, Claude 3.5 Haiku, and Claude 3.7 Sonnet.
- All methods are evaluated under fixed API budgets to ensure fair comparison.The total API compute cost was on the order of tens of thousands of US dollars.
- Task-appropriate metrics include exact match, success and progress rates, cumulative accuracy, and robustness over task streams.Results are averaged across multiple task instances and trajectories.
- The experiments use standard Python tooling, official vendor SDKs, PyTorch 2.7.1, and Weights & Biases for utilities and evaluation.
- The authors used ChatGPT only for stylistic refinement, while developing and verifying the scientific content themselves.Its use covered clarity, grammar, and readability rather than scientific ideas, analyses, experiments, or conclusions.
I. Limitations
The study’s scope is constrained by model coverage, API and budget limits, and an emphasis on textual, goal-oriented tasks. The reported tables organize cross-environment and cross-dataset comparisons but do not themselves expand that scope.
- Budget and API limits restrict evaluation to a selected set of strong LLMs rather than all available models.
- Open-weight and multilingual evaluations could further test the generality of the findings.
- The benchmark primarily emphasizes textual and goal-oriented tasks rather than richer multimodal or real-world environments.Extending evaluation to those settings would provide a more complete picture of continual memory evolution.
- Table 4 reports success and progress rates across four embodied reasoning benchmarks, including averaged values across datasets.
- Table 5 compares diverse memory architectures across models using Exact Match and API/Acc metrics.The table separates zero-shot, agentic, adaptive, procedural, and proposed memory methods; unreliable methods are omitted.