Source-linked AI summary
Memp: Exploring Agent Procedural Memory
Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, Ningyu Zhang
TL;DR
LLM agents need procedural memory that can be learned from experience rather than manually engineered or fixed in parameters. Memp studies how agents construct, retrieve, and update procedural memories, with evaluations showing improved task success and efficiency, including on ALFWorld. The framework supports continual refinement, but remains limited by vector-based retrieval with manual keys and benchmark-supplied rewards.
Problem
LLM agents face brittle procedural memory that is manually engineered or entangled in static parameters, despite complex tasks sharing reusable structure.
Method
Memp treats procedural memory as a core optimization target and studies strategies for constructing, retrieving, and updating memories from prior task experience.
Results
Memp achieves the best ALFWorld performance among compared baselines, with the highest dev and test success rates and fewest steps.
Takeaways & Limitations
Memp enables agents to distill, reuse, and refine past experiences across long-horizon tasks while supporting continual learning and robust generalization.
Takeaways & Limitations
Retrieval is restricted to vector-similarity search with manually crafted keys, and task-success assessment depends on explicit benchmark-supplied rewards.
Abstract
from arXiv · showhide
Large Language Models (LLMs) based agents excel at diverse tasks, yet they suffer from brittle procedural memory that is manually engineered or entangled in static parameters. In this work, we investigate strategies to endow agents with a learnable, updatable, and lifelong procedural memory. We propose Memp that distills past agent trajectories into both fine-grained, step-by-step instructions and higher-level, script-like abstractions, and explore the impact of different strategies for Build, Retrieval, and Update of procedural memory. Coupled with a dynamic regimen that continuously updates, corrects, and deprecates its contents, this repository evolves in lockstep with new experience. Empirical evaluation on TravelPlanner and ALFWorld shows that as the memory repository is refined, agents achieve steadily higher success rates and greater efficiency on analogous tasks. Moreover, procedural memory built from a stronger model retains its value: migrating the procedural memory to a weaker model can also yield substantial performance gains. Code is available at https://github.com/zjunlp/MemP.
1 Introduction
Long-horizon agents face costly, failure-prone execution despite increasingly sophisticated capabilities, motivating reusable procedural memory. Memp studies how to construct, retrieve, and update such memory so agents can reuse experience and improve across tasks.
- Motivation: Long-horizon tasks require many steps and can be derailed by network glitches, interface changes, or shifting data schemas.Restarting from scratch makes failures especially costly for current agents.
- Motivation: Procedural knowledge in contemporary agents is often handcrafted, encoded in brittle prompts, or entangled in expensive-to-update model parameters.This limits how agents retain and reuse experience across trajectories.
- Memp: Memp treats procedural memory as a first-class optimization object and systematically examines its construction, retrieval, and updating strategies.The framework explores trajectory- or guideline-based construction, key-based retrieval, and addition, validation, reflection, and discarding updates.
- Results: On TravelPlanner and ALFWorld, procedural-memory construction and retrieval improve task accuracy while reducing steps and token consumption relative to isolated task solving.The reported gains arise when agents exploit prior experience at test time.
- Results: Memory updates let agents build and refine procedural knowledge while acting, producing a continual, almost linear mastery of the task.Ablations also report scaling with task complexity and transfer to related tasks.
2 Related Works
Related work studies memory systems and learning from experience as ways to improve agent coherence, adaptability, and decision-making. However, procedural memory still lacks systematic analysis of how it should be constructed, retrieved, and updated.
- Memory in Language Agents: Language-agent memory systems span end-to-end, external, and hierarchical designs that store information across short-term, episodic, and long-term timescales.They use formats such as vector embeddings and semantic search, alongside updating and forgetting strategies.
- Procedural Memory: Procedural-memory methods automate repetitive tasks, decisions, and interaction patterns, but existing work lacks systematic analysis of their construction, retrieval, and updating.The gap remains despite methods such as Voyager, AWM, and AutoManual.
- Learning from Experience: Learning from experience improves agent decision-making through interaction and reuse of past experiences across tasks and dynamic scenarios.This direction also aims to reduce manual programming needs.
3 Preliminary
Agents operating in complex environments generate trajectories and rewards, but repeated exploration wastes time and tokens when prior procedural knowledge is not transferred. The framework models procedural memory as a library that is built from experience, retrieved for similar tasks, and dynamically updated using execution feedback.
- Agent-environment interaction: Complex multistep agent interactions can be modeled as Markov Decision Processes, where policies select actions, environments transition states, and trajectories record the interaction.A trajectory is represented as τ = (s0, a0, o1, s1, a1, o2, . . . , sT ).
- Motivation: Exploratory actions provide useful context but impose high test-time costs, especially when similar tasks repeat within the same environment.Agents may redundantly spend actions or tokens understanding an environment and task they have partly encountered before.
- Procedural memory: A procedural memory module transforms the conventional policy π(at|st) into πmp(at|st), allowing experience from earlier sequential tasks to reduce repetitive exploration.For a trajectory τ and reward r, a builder B constructs procedural memory mp.
- Framework overview: The framework organizes procedural memory into Build, Retrieve, and Update operations for encoding, forming, and modifying memories from new experiences.These modules are analyzed as distinct parts of the procedural memory framework.
- Retrieval: The procedural memory library stores memories acquired over completed tasks, and retrieval selects a memory associated with a task most similar to the new task.The experiments use cosine similarity for task embeddings in retrieval.
- Update: As tasks accumulate, updating must dynamically add, delete, modify, and retrieve memories according to execution context and feedback.The update function takes current memory and task execution feedback, including success, failure, and performance metrics.
4 Experiment
Memp evaluates procedural-memory construction, retrieval, and updating across TravelPlanner and ALFWorld. The experiments compare memory formats and retrieval strategies, showing that procedural memory improves performance and that continual, error-guided updating is especially effective.
- 4.1 Experimental Settings: Experiments adapt TravelPlanner, a tool-use planning benchmark, and ALFWorld, a household-task benchmark with textual environment feedback.
- 4.2 Memory Storage & Retrieval: Procedural memory is stored either as complete trajectories, abstract scripts, or a combination of both, then appended to task context when retrieved.
- 4.2 Memory Storage & Retrieval: Scripts generalize better to different test tasks, trajectories perform better on tasks resembling completed ones, and combining both achieves the best performance.
- 4.2 Memory Storage & Retrieval: Query-based and AveFact retrieval outperform random sampling by using semantic context or extracted task features for more precise memory matching.
- 4.3 Memory Update: By the final task group, reflexion-based updating exceeds the second-best strategy by +0.7 points and reduces the trajectory by 14 steps.
- 4.4 Comparison: Memp achieves the highest dev and test success rates on ALFWorld while requiring the fewest steps among the compared GPT-4o baselines.
5 Analysis
The analysis examines how procedural memory changes agent efficiency, transfers across model sizes, and scales with retrieval quantity. It finds that memory reduces trial-and-error, can improve a smaller model, and helps until excessive retrieval causes interference.
- Efficiency: Procedural Memory reduces trial-and-error by guiding object locations and consecutive actions, allowing complex tasks to finish in fewer steps.
- Retrieval Scaling: As more procedural memories are retrieved, performance improves steadily before plateauing, but excessive retrieval can reduce performance through context-length pressure and inaccurate memories.
6 Conclusion and Future Work
Memp makes procedural memory a core optimization target for LLM-based agents, systematically studying how construction, retrieval, and updating affect performance. It supports agents in distilling, reusing, and refining past experiences while enabling continual learning and a continuous self-improvement loop.
- Memp elevates procedural memory to a core optimization target in LLM-based agents.
- The framework systematically studies strategies for constructing, retrieving, and updating procedural memory.
- Memp enables agents to distill, reuse, and refine past experiences across diverse, long-horizon tasks.
- Future Work: The proposed lifecycle continuously executes tasks, self-assesses completion, builds memories, and progresses.
Limitations
Memp remains constrained by its retrieval design and dependence on explicit benchmark rewards. These limitations restrict retrieval diversity and make task-success assessment difficult in real-world settings with sparse or absent rewards.
- Memp restricts retrieval to vector-similarity search with manually crafted keys, excluding methods such as BM25.
- Memp depends on explicit benchmark-supplied reward signals and cannot yet judge task success in real-world settings where rewards are sparse or absent.
A Use of AI Assistant
The authors state that LLMs were used only as an assisted tool for refining wording and sentence structure during paper writing. They report no other reliance on LLMs in this work.
- LLMs were used solely to refine wording and sentence structure during paper writing.
- The authors report no other reliance on LLMs beyond the documented writing assistance.
B Case Study
The paper provides detailed case studies in Figure 6.
- Detailed case studies are provided in Figure 6.
C Evaluation Details
The evaluation uses environment- or gold-standard-based scoring, while Figure 6 compares trajectories with and without procedural memory for efficiency.
- ALFWorld task completion is evaluated by the execution environment after completion or the maximum number of execution steps.
- TravelPlanner experiments use a two-stage test-set procedure that obtains a travel trajectory and final planner before GPT-4o converts the plan to JSON.
- 685 tokens are saved and the process is shortened by 9 steps with procedural memory in the Figure 6 comparison.