Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution

Zouying Cao; Jiaji Deng; Li Yu; Weikang Zhou; Zhaoyang Liu; Bolin Ding; Hai Zhao

Source-linked AI summary

Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution

Zouying Cao, Jiaji Deng, Li Yu, Weikang Zhou, Zhaoyang Liu, Bolin Ding, Hai Zhao

arXiv:2512.10696v2Dec 11, 2025cs.AIcs.CL

TL;DR

Procedural-memory systems need to move beyond static accumulation toward reusable, task-grounded, and continuously optimized experience. ReMe implements this through multi-faceted distillation, context-adaptive reuse, and utility-based refinement, and experiments report state-of-the-art performance, including gains for memory-equipped Qwen3-8B over memoryless Qwen3-14B.

Problem
Existing procedural-memory frameworks predominantly use passive, static storage, while an ideal system must extract reusable knowledge, adapt it to tasks, and optimize its memory pool over time.
Method
ReMe combines multi-faceted experience distillation, context-adaptive reuse, and utility-based refinement across the procedural-memory lifecycle.
Results
ReMe achieves state-of-the-art performance on BFCL-V3 and AppWorld; Qwen3-8B with ReMe outperforms memoryless Qwen3-14B by 8.83% in Avg@4 and 7.29% in Pass@4.
Takeaways & Limitations
The results suggest that self-evolving memory can provide a resource-efficient pathway for lifelong learning in LLM agents.
Takeaways & Limitations
ReMe retrieves experiences only once at task start and relies primarily on an LLM-as-judge process for experience validation.

Abstract

from arXiv · show

Procedural memory enables large language model (LLM) agents to internalize "how-to" knowledge, theoretically reducing redundant trial-and-error. However, existing frameworks predominantly suffer from a "passive accumulation" paradigm, treating memory as a static append-only archive. To bridge the gap between static storage and dynamic reasoning, we propose $\textbf{ReMe}$ ($\textit{Remember Me, Refine Me}$), a comprehensive framework for experience-driven agent evolution. ReMe innovates across the memory lifecycle via three mechanisms: 1) $\textit{multi-faceted distillation}$, which extracts fine-grained experiences by recognizing success patterns, analyzing failure triggers and generating comparative insights; 2) $\textit{context-adaptive reuse}$, which tailors historical insights to new contexts via scenario-aware indexing; and 3) $\textit{utility-based refinement}$, which autonomously adds valid memories and prunes outdated ones to maintain a compact, high-quality experience pool. Extensive experiments on BFCL-V3 and AppWorld demonstrate that ReMe establishes a new state-of-the-art in agent memory system. Crucially, we observe a significant memory-scaling effect: Qwen3-8B equipped with ReMe outperforms larger, memoryless Qwen3-14B, suggesting that self-evolving memory provides a computation-efficient pathway for lifelong learning. We release our code and the $\texttt{reme.library}$ dataset to facilitate further research.

1 Introduction

ReMe addresses the limitations of passive procedural-memory storage by distilling fine-grained experiences, adapting their reuse to task contexts, and refining the memory pool. Experiments report state-of-the-art performance and show that memory-equipped Qwen3-8B can outperform memoryless Qwen3-14B.

Motivation: Procedural memory aims to help agents reuse how-to knowledge from past interactions, reducing redundant trial-and-error without model retraining.Agents accumulate problem-solving experiences to navigate novel scenarios.
Design Requirements: An effective procedural-memory system requires high-quality extraction, task-grounded utilization, and progressive optimization of its experience pool.These criteria target reusable knowledge, context-specific adaptation, and removal of outdated entries.
Limitations of Existing Frameworks: Existing frameworks treat memory as passive accumulation, using raw trajectories or whole-trajectory workflows as largely static experiences.This creates coarse-grained representations and limits dynamic memory use.
ReMe Framework: ReMe combines multi-faceted distillation, context-adaptive reuse, and utility-based refinement to create a feedback-driven procedural-memory lifecycle.It recognizes success patterns, analyzes failures, generates comparative insights, adapts retrieved experiences, and prunes low-utility entries.
Results: 8.83% in Avg@4 and 7.29% in Pass@4 are the reported average gains for ReMe-equipped Qwen3-8B over memoryless Qwen3-14B.Experiments on BFCL-V3 and AppWorld report state-of-the-art performance and a memory-scaling effect.
Resources: The released reme.library dataset contains fine-grained procedural memories with structured success patterns and failure lessons for future research.The resource is intended for studying procedural memory and optimizing memory-augmented agents.

2 Related Works

Related work uses memory systems to enhance LLM-agent reasoning and training efficiency, but existing approaches do not strategically remove harmful or degraded experiences.

Memory-Enhanced LLM Agents: Memory-enhanced agents use parametric or non-parametric memory to retain knowledge and enrich task contexts without necessarily modifying model parameters.Examples include world-knowledge models, induced task workflows, and user-preference memories.
Limitations: Existing methods neglect strategic experience removal because harmful experiences can persist and initially helpful experiences can degrade over time.This limitation motivates memory-management mechanisms that update experience quality over time.

3 Methodology

ReMe implements procedural memory as a closed loop of experience acquisition, context-adaptive reuse, and utility-based refinement. It distills structured insights from trajectories, adapts retrieved experiences to new tasks, and selectively updates the pool to retain useful knowledge.

Framework overview: ReMe alternates experience acquisition, reuse, and refinement to build, apply, and update an experience pool after task execution.The framework summarizes trajectories, recalls relevant experiences for new tasks, and selectively adds or removes experiences.
Experience acquisition: The summarizer extracts structured experiences through success-pattern recognition, failure analysis, and comparative analysis of successful and unsuccessful trajectories.These analyses identify effective strategies, pitfalls, and differences between effective and ineffective attempts.
Experience acquisition: Validated experiences are deduplicated and indexed by usage-scenario embeddings in a vector database for future retrieval.The pool stores compact, reusable experiences after validation and similarity-based deduplication.
Experience reuse: Retrieval ranks top-K experiences by task similarity, while reranking and rewriting adapt them into cohesive, task-specific guidance.The reuse pipeline evaluates relevance against the current task’s constraints and objectives before reorganizing retrieved context.
Experience refinement: The resulting self-evolving pool retains high-quality experiences while adapting to changing task demands.This refinement mechanism addresses the inability of static pools to adapt to task-distribution shifts and improving model capability.
Experience refinement: ReMe uses selective addition and failure-aware reflection to incorporate successful lessons while discarding unsuccessful exploratory insights.Full addition can underperform selective addition; failure-derived lessons enter memory only when a subsequent trial succeeds.
Experience refinement: Experiences whose average utility falls below β are deleted after at least α retrievals, using recall frequency and successful-task contributions as evidence.Utility increases when recalling an experience contributes to successful task completion, and removal targets frequently retrieved experiences that fail to improve performance.

4 Experiments

Experiments evaluate ReMe on BFCL-V3 and AppWorld against no-memory and memory baselines, with results spanning main comparisons, generalization, and ablations. ReMe consistently improves performance, with adaptive refinement, fine-grained extraction, selective experience addition, and usage-scenario retrieval contributing to the observed gains.

Main Results: ReMe achieves the highest average task success rate across three model sizes, consistently outperforming No Memory and competitive memory baselines.The evaluation covers BFCL-V3 and AppWorld.
Main Results: 7.29% Pass@4 and 8.83% Avg@4 are the average improvements of Qwen3-8B with ReMe over the No Memory baseline.These results are reported across the evaluated benchmarks.
Main Results: 55.03% vs. 54.65% Pass@4 shows Qwen3-8B with ReMe (dynamic) surpassing vanilla Qwen3-14B without memory.Qwen3-14B with ReMe also exceeds Qwen3-32B without memory: 44.66% vs. 40.89% Avg@4 and 63.71% vs. 61.52% Pass@4.
Main Results: ReMe (dynamic) consistently outperforms ReMe (fixed) across model sizes and benchmarks, while also tending to reduce performance standard deviation across runs.The reported stability improvement is particularly evident for larger models.
Ablation Studies: Keypoint-level experiences produce substantially larger gains than trajectory-level experiences, indicating that fine-grained summaries transfer knowledge more effectively.Trajectory-level experiences show only minor progress over the No Memory baseline.
Ablation Studies: Selective addition improves BFCL-V3 by 3.50% Avg@4 and 2.66% Pass@4, while failure-aware reflection and utility-based deletion provide further gains.The ablation uses Qwen3-8B with thinking disabled in the ReMe(fixed) setting.

5 Conclusion

ReMe is presented as a dynamic procedural-memory framework that turns prior trajectories into structured knowledge for strategic experience reuse. Its components and experiments support agent evolution without relying only on blind trial-and-error.

ReMe distills structured knowledge from prior trajectories at a fine-grained level for strategic experience reuse.This is intended to reduce interference associated with coarse-grained approaches.
ReMe uses experience refinement to maintain a high-quality experience pool for agent evolution.
Ablation studies highlight the value of each core ReMe component.

Limitations

The paper identifies limitations in ReMe’s retrieval, validation, and summarization strategies. These limitations define areas for future improvement in procedural-memory management.

ReMe retrieves experiences only once at the beginning of each task, limiting retrieval flexibility.The authors suggest more context-aware retrieval for adaptive knowledge utilization.
The experience validation process relies primarily on an LLM-as-judge approach, which may miss nuanced quality and relevance issues.The paper proposes exploring more sophisticated validation techniques.
Larger-scale summarizers produce greater gains in agent reasoning, indicating that small-model summarization remains an improvement target.The authors attribute this to stronger summarization capability and propose advanced strategies with small models.

B Experimental Details

This section outlines the experimental setup used to reproduce and evaluate ReMe. It covers datasets, metrics, baselines, and implementation details.

The experiments include benchmark datasets for evaluating ReMe.
The setup specifies evaluation metrics and baseline methods for comparison.
Implementation details are provided to support reproduction of the reported results.

B.1 Datasets and Evaluation Metrics

The experiments use BFCL-V3 and AppWorld to evaluate tool-augmented agents, with task success measured through benchmark-specific correctness or completion criteria. The setup also compares multiple models and memory systems.

Datasets and Evaluation Metrics: BFCL-V3 evaluates function calling and tool use in multi-turn and multi-step scenarios using AST matching and executable testing.It contains over 1,800 test tasks spanning Python, Java, JavaScript, and complex interactions such as parallel function calls.
Datasets and Evaluation Metrics: A task is successful when the agent makes the necessary function calls correctly and produces the expected outputs.
Datasets and Evaluation Metrics: AppWorld evaluates function calling and interactive coding across nine applications and 457 APIs using state-based unit tests.The reported metric is Task Goal Completion, the percentage of tasks passing all evaluation tests.
Experimental Setup: The study compares Qwen3 models at 8B, 14B, and 32B sizes, enabling analysis of memory-enhanced performance across model scales.Qwen3 thinking mode is enabled for BFCL-V3 and disabled for AppWorld.
Experimental Setup: Additional BFCL-V3 experiments evaluate several other LLMs with and without ReMe to test generalizability.The listed models include GPT-4.1, o4-mini, Qwen3-Max-Preview, Kimi-K2-Thinking, DeepSeek-V3.2, and GLM-4.7.
Experimental Setup: The BFCL-V3 comparison is reported with the Avg@4 percentage metric.

B.4 Implementation Details

ReMe builds experiences from sampled trajectories and retrieves them by matching new task queries to indexed usage scenarios. Its implementation also compares experience granularity and evaluates performance across models and benchmark settings.

ReMe samples each task query 8 times, ranks trajectories by reward, and selects the highest- and lowest-scoring examples for experience acquisition.
Experience acquisition recognizes success patterns, analyzes failures, and generates comparative insights, followed by LLM-based validation of generated experiences.
Retrieved experiences are matched to new task queries through usage-scenario fields and cosine similarity between their vector representations.
ReMe emphasizes keypoint-level experiences that retain critical actions while omitting less relevant steps, unlike trajectory-level experiences that preserve exhaustive procedural details.
ReMe consistently outperforms the No Memory baseline across GPT-4.1, o4-mini, Qwen3-Max, Kimi-K2, DeepSeek-V3.2, and GLM-4.7 backbones.

D.2 Results on Larger-scale Benchmark

ReMe remains effective on a larger BFCL-V3 evaluation comprising 600 tasks across Missing Parameters, Missing Functions, and Long Context categories.

600 tasks were used to evaluate ReMe on BFCL-V3 Missing Parameters, Missing Functions, and Long Context categories using Pass@4.
The experience pool was constructed from 50 tasks in the base multi-turn category, yet ReMe still performed well on the larger benchmark.

D.3 Retrieval Key Analysis

Retrieval-key analysis shows that LLM-generated usage-scenario keys are more reliable than simple task-query or keyword indexing across model scales. A trajectory comparison illustrates how retrieved experience can guide correct task execution.

Table 9 compares task query, generalized query, query keywords, and usage scenario retrieval keys across Qwen3-8B, Qwen3-14B, and Qwen3-32B.
Raw task queries and query keywords generally yield lower performance, whereas usage-scenario keys achieve the highest or near-highest Avg@4 and Pass@4 scores across models.
On a BFCL-V3 purchasing task, retrieved experience guided the agent to obtain real-time Apple pricing before placing an order, avoiding the memoryless agent’s fabricated price.