PaperlayerView exact revision on arXiv

Source-linked AI summary

MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

Shengtao Zhang, Jiaqian Wang, Ruiwen Zhou, Junwei Liao, Yuchen Feng, Zhuo Li, Yujie Zheng, Weinan Zhang, Ying Wen, Zhiyu Li, Feiyu Xiong, Yutao Qi, Bo Tang, Muning Wen

arXiv:2601.03192v2cs.CL

TL;DR

AI agents need to improve after deployment without the cost and forgetting risks of weight updates, but conventional memory retrieval relies mainly on semantic similarity. MemRL decouples a frozen language model from trainable episodic memory, using value-aware retrieval and reward-based utility updates; across four benchmarks, it reports consistent superiority over baselines and enables runtime adaptation without weight modification.

  • Problem

    Fine-tuning is computationally costly and prone to catastrophic forgetting, while passive semantic retrieval can select noisy experiences instead of useful strategies.

  • Method

    MemRL uses an Intent-Experience-Utility memory structure, Two-Phase Retrieval, and runtime reinforcement-learning updates of memory Q-values while keeping the language-model backbone frozen.

  • Results

    MEMRL consistently outperforms baselines across HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench, exceeding MemP by an average of +3.8% in Cumulative Success Rate.

  • Takeaways & Limitations

    The results support non-parametric memory evolution as a way for agents to adapt from interaction while preserving a stable backbone and avoiding weight updates.

  • Takeaways & Limitations

    Step-wise updates may introduce high-variance noise in long-horizon trajectories, and performance may drift toward reflection-like behavior when task similarity is low.

Abstract

from arXiv · show

The hallmark of human intelligence is the self-evolving ability to master new skills by learning from past experiences. However, current AI agents struggle to emulate this self-evolution: fine-tuning is computationally expensive and prone to catastrophic forgetting, while existing memory-based methods rely on passive semantic matching that often retrieves noise. To address these challenges, we propose MemRL, a non-parametric approach that evolves via reinforcement learning on episodic memory. By decoupling stable reasoning from plastic memory, MemRL employs a Two-Phase Retrieval mechanism to filter noise and identify high-utility strategies through environmental feedback. Extensive experiments on HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench demonstrate that MemRL significantly outperforms state-of-the-art baselines, confirming that MemRL effectively reconciles the stability-plasticity dilemma, enabling continuous runtime improvement without weight updates. Code is available at https://github.com/MemTensor/MemRL.

1. Introduction

MemRL addresses the difficulty of runtime self-evolution by decoupling stable model reasoning from plastic episodic memory. It uses non-parametric reinforcement learning to select and update useful experiences without changing model weights.

  • Fine-tuning is costly and vulnerable to catastrophic forgetting, while semantic-similarity retrieval can return noisy memories without measuring utility.
  • MemRL frames continuous post-deployment improvement as learning with a frozen backbone, preserving stability while adapting to new tasks.
  • The method stores Intent-Experience-Utility triplets and turns retrieval into an active decision process guided by learned Q-values.
  • Two-Phase Retrieval and Utility-Driven Update use environmental feedback to distinguish high-value experiences from semantically similar noise.
  • MEMRL was evaluated on HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench, with reported superiority over baselines and stronger gains in exploration-heavy environments.
  • The authors analyze MEMRL’s stability and report that it minimizes forgetting while maintaining task integrity.

2. Related Works

Related work spans runtime learning, agentic memory, and reinforcement learning for improving language-model agents. MemRL builds on these directions by treating memory reuse as value-based decision-making guided by environmental rewards.

  • Runtime learning emphasizes post-deployment improvement through interaction streams while keeping the backbone frozen for stability and efficiency.
  • Reinforcement-learning approaches improve LLMs through human-feedback rewards, rule-based verifiers, and interaction signals for tool use and action decisions.
  • Memory-augmented agents have progressed from static RAG toward reflection, hierarchical management, structured indexing, and dynamic memory governance.
  • Its illustrative MDP example shows a successful experience from one intent later being reused by another intent to produce a successful outcome.
  • MemRL formulates retrieval as a value-based decision process and learns Q-values from environmental rewards to distinguish high-value experiences.

3. Problem Formulation

MemRL formulates memory-augmented generation as a memory-based Markov Decision Process and optimizes retrieval rather than the frozen language model. Its retrieval policy learns which memories produce useful downstream actions from environmental feedback.

  • 3. Problem Formulation: The M-MDP formulation represents memory-augmented generation as a sequential decision process whose retrieval outcomes and memory evolution unfold across time.
  • 3.1. Memory-Augmented Agent Policy: The M-MDP tuple includes state, action, transition dynamics, reward, discount factor γ, and an evolving bank of past experiences.
  • 3.1. Memory-Augmented Agent Policy: At each step, the agent uses its current state and memory to generate an action maximizing expected reward, with the joint policy marginalizing over retrieved items.
  • 3.1. Memory-Augmented Agent Policy: The retrieval policy selects memory contexts while the frozen LLM generates actions, making retrieval an active process that accounts for functional utility.
  • 3.1. Memory-Augmented Agent Policy: Fixed similarity retrieval matches semantic content but does not determine whether a memory actually leads to a successful outcome.
  • 3.2. Non-Parametric Reinforcement Learning: MEMRL directly optimizes retrieval within memory space by mapping the M-MDP to an Intent-Experience-Utility triplet.
  • 3.2. Non-Parametric Reinforcement Learning: Q(s, m) estimates the expected utility of the subsequent action when memory m augments the agent, allowing retrieval to favor strategically helpful memories over irrelevant noise.
  • 3.2. Non-Parametric Reinforcement Learning: Environmental rewards update memory utilities through TD or Monte Carlo-style rules, with the simplified terminal-state update designed to balance complexity and performance.

4. MEMRL

MEMRL enables frozen LLMs to self-evolve by optimizing memory use rather than model weights. It combines structured episodic memory, value-aware retrieval, feedback-based utility updates, and a stability analysis.

  • MEMRL optimizes the retrieval policy within evolving memory while keeping the LLM weights frozen.
  • 4.1. The Intent-Experience-Utility Triplet: The Intent-Experience-Utility triplet stores an intent, raw experience, and learned utility estimating expected return for similar intents.The utility serves as the critic in reinforcement learning.
  • 4.2. From Semantic Recall to Value-Aware Selection: Two-Phase Retrieval first recalls semantically consistent experiences, then selects top-k2 candidates using similarity and utility.If no candidates are recalled, the frozen LLM performs exploration.
  • 4.2. From Semantic Recall to Value-Aware Selection: The selection score balances normalized similarity and utility through λ, filtering semantically similar memories with low historical utility.Normalization and strict similarity thresholds are described as essential for noise filtering and stability.
  • 4.3. Non-Parametric RL on Memory: Runtime utility updates use environmental rewards to refine Q-values, while summarized trajectories expand memory without changing LLM parameters.The update moves Qnew toward empirical expected returns for experiences under similar intents.
  • 4.4. Theoretical Stability Analysis: Under a frozen inference policy and stationary task distribution, the analysis states that utility estimates are unbiased and variance-bounded and that the system converges to a stable point.The stability analysis associates the limiting formulation with global stability and prevention of catastrophic forgetting.

5. Experiments

MEMRL is evaluated across runtime learning, transfer, ablations, and mechanism analyses using Success Rate and Cumulative Success Rate. It consistently improves performance and stability by selecting memories through learned utility rather than similarity alone.

  • Main Results: MEMRL surpasses MemP by an average of +3.8% in Cumulative Success Rate across runtime-learning benchmarks.The gains reach +6.2% on ALFWorld and OS tasks and +3.6% on HLE.
  • Main Results: MEMRL outperforms MemP by an average of +2.8% in Success Rate when its memory bank transfers to held-out tasks.The advantage is +5.8% on ALFWorld and +2.6% on OS tasks.
  • Runtime RL: In OS interaction, MEMRL develops smoother learning and a widening Cumulative Success Rate gap over MemP and RAG as training progresses.The dashed curves represent Cumulative Success Rate, while solid curves represent Epoch Success Rate.
  • Runtime RL: λ = 0.5 achieves the best trade-off between semantic relevance and Q-value helpfulness, while extreme weighting causes plateaus or volatility.Pure semantic retrieval cannot filter functional distractors, whereas excessive RL weighting induces context detachment.
  • Retrieval Scope: Cross-task retrieval improves structured-environment performance by +9.0% on OS-Agent and +5.1% on ALFWorld, but HLE shows no advantage over single-task reflection.On HLE, the single-task baseline scores 0.610 versus MEMRL’s 0.606, attributed to low internal semantic similarity of 0.186.
  • Mechanism Analysis: The critic’s Q-values correlate with empirical success at Pearson r = 0.861, while approximately 12% of high-Q memories are failures that may contain strategically useful near-misses.Success rates rise from 21.5% in the lowest-confidence bin to 88.1% in the highest.
  • Stability Analysis: MEMRL achieves a mean Forgetting Rate of 0.041 versus MemP’s 0.051, while removing normalization and similarity gating raises the rate to 0.073.The results associate strict filtering with more stable self-evolution.

6. Limitations and Conclusion

MEMRL’s runtime dynamics expose limitations involving noisy long-horizon updates, ambiguous credit assignment, and reduced effectiveness when task similarity is low. The paper concludes that value-based memory retrieval supports self-evolution and generalization without weight updates.

  • Limitations: Step-wise updates may introduce high-variance noise in long-horizon trajectories, motivating multi-step updates or periodic memory consolidation.
  • Limitations: Utility updates face credit-assignment ambiguity when multiple experiences are referenced, motivating more precise attribution methods.
  • Limitations: When task similarity is low, performance may drift toward reflection-like behavior, making a sufficiently diverse yet relevant experience base important for deployment.
  • Conclusion: MEMRL treats retrieval as value-based decision-making and uses Intent-Experience-Utility triplets with Monte Carlo-style updates to distinguish useful strategies from semantic noise.
  • Conclusion: Extensive evaluations report that MEMRL significantly outperforms baselines in runtime adaptation and generalization without updating model weights.

Impact Statement

MEMRL is presented as a value-reinforced memory retrieval mechanism for improving long-term reasoning in LLM agents. By optimizing memory retrieval rather than model weights, it is described as potentially reducing deployment overhead and environmental impact.

  • MEMRL optimizes memory retrieval to enhance the long-term reasoning capabilities of LLM agents.
  • The paper suggests that retrieval optimization may reduce computational overhead in large-scale agent deployments and potentially lower AI infrastructure’s environmental impact.

A.1. Stability Analysis

The stability analysis models MEMRL’s frozen inference policy and evolving memory utilities under stationary reward and task-distribution assumptions. It shows convergence of utility estimates, bounded variance, and global stability through alternating retrieval-policy and value updates.

  • Setup and assumptions: MEMRL performs runtime learning with constant-step-size updates while keeping the inference policy and evaluator criteria fixed.
  • Setup and assumptions: The analysis assumes tasks are sampled from a stationary distribution over a fixed dataset, making each task-memory pair’s expected reward time-invariant.
  • Expected convergence: For a fixed task-memory pair updated infinitely often, expected utility converges to the true mean reward β(s, m).
  • Expected convergence: The estimation error decays geometrically as E[e_t] = (1 − α)^t e_0, yielding exponential convergence when 0 < α < 1.
  • Variance and global stability: With finite reward variance, constant-step-size updates keep utility variance bounded and filter high-frequency noise while tracking expected memory effectiveness.
  • Global stability: GEM-style alternation between Phase-B policy improvement and utility updates increases a shared objective and converges to a stationary retrieval policy.

B.1. Detailed Analysis of Forgetting Dynamics

On HLE, MEMRL maintains lower forgetting than MemP and its ungated ablation, while gains are especially pronounced for sequential tasks and can transfer across inference models. The analyses associate these benefits with trajectory-level verification, structural repetition, and portable procedural patterns.

  • Forgetting dynamics: MEMRL maintains a consistently low forgetting rate on HLE, whereas MemP’s rate rises gradually with episode count.
  • Forgetting dynamics: 0.073 mean forgetting rate is reported for the ablation without normalization and similarity gating, whose spikes indicate noisy strategy reinforcement.
  • Task structure: ALFWorld gains +6.2% Points(pp), exceeding BigCodeBench’s +2.5% pp, with larger benefits on multi-step sequential tasks.
  • Task structure: MEMRL propagates final trajectory rewards backward to memory utilities, helping filter policies that match initial instructions but fail at later steps.
  • Similarity and generalization: HLE improves from 0.357 to 0.570 (∆= +0.213) despite similarity of 0.186, indicating runtime gains can arise through specific knowledge acquisition rather than only pattern generalization.
  • Cross-model transfer: A memory bank trained with Gemini-3-pro transfers zero-shot across Qwen3-235B, GPT-5.2(High), and Gemini-3-flash without fine-tuning.
  • Cross-model transfer: Qwen3-235B improves by over 3× from 0.150 to 0.531, while Gemini-3-flash rises from 0.347 to 0.583.
  • Cross-model transfer: The transferred results suggest that MEMRL captures model-agnostic problem-solving patterns rather than model-specific artifacts.

C.2. Multi-Task Memory Merging and Interference Analysis

The memory-merging experiment tests whether a unified bank preserves performance across OS and database tasks. Results show negligible cross-task interference under frozen-backbone evaluation.

  • Memory merging: The experiment merged finalized OS and database memory banks and evaluated both tasks without further training or fine-tuning.The unified bank was defined as MUnified = MOS ∪ MDB.
  • Interference results: 0.960 DB Task performance remained identical after memory merging.
  • Interference results: OS Task performance changed only marginally, from 0.788 to 0.784, under the unified memory bank.
  • Interference mechanism: Two-Phase Retrieval limits interference by using semantic filtering to exclude irrelevant cross-task memories before utility-based selection.OS commands and SQL queries occupy largely orthogonal semantic spaces.
  • Evaluation setup: Baselines were evaluated with the same frozen-backbone setting to isolate memory and retrieval mechanisms.

D.2. Benchmark Datasets

The evaluation spans code generation, OS and database interaction, embodied decision-making, and multidisciplinary reasoning. The section also specifies reproducibility settings, data partitions, and transfer-learning controls.

  • Benchmark scope: Four benchmarks cover code generation, OS interactions, embodied decision-making, and multidisciplinary reasoning.The suite comprises BigCodeBench, Lifelong Agent Bench, ALFWorld, and Humanity’s Last Exam.
  • BigCodeBench: BigCodeBench-Instruct evaluates complete functional code synthesis using diverse third-party libraries.
  • Lifelong Agent Bench: Lifelong Agent Bench tests adaptation to new OS and database tools over long horizons without forgetting previous skills.
  • ALFWorld: ALFWorld evaluates multi-step plans for textual navigation and manipulation tasks in a simulated household environment.
  • Humanity’s Last Exam: Humanity’s Last Exam stress-tests multidisciplinary reasoning across mathematics, humanities, and sciences.
  • Reproducibility and partitioning: The experiments provide model, hyperparameter, environmental, and data-partition details for reproducibility.Random splits use seed 42, while ALFWorld transfer evaluates 140 novel instances of seen task types using 3,553 tasks as memory context.

E.4. Model Selection and Performance Analysis

Models are matched to task difficulty to avoid unusable feedback and ceiling effects, while experiments examine robustness across scales, tasks, transfer settings, and runtime overhead. MEMRL shows broad gains but retains limitations in long-horizon updates, task similarity, feedback quality, and deployment scope.

  • Model selection: Model selection balances initial competence and headroom, avoiding too few successful trajectories for utility estimation and performance saturation.On HLE, GPT-4o-mini may begin near 4% success, producing too little feedback for stable utility estimation.
  • Cross-model robustness: MEMRL remains effective across backbone scales, and cross-model transfer indicates that procedural memory is portable across backbones.
  • Runtime performance: 12.6% points higher average Last Epoch Success Rate than No Memory and 11.2% points higher average Cumulative Success Rate than MemP were achieved in runtime learning.
  • Transfer performance: 10.6% points higher average Success Rate than No Memory and 6.1% points higher than MemP were achieved in transfer learning.
  • Inference cost: MEMRL achieves these gains without increasing the inference budget because retrieval and Q-value updates, rather than added context, drive its complexity.The cited passage reports similar 32K token usage.
  • Algorithmic overhead: MEMRL’s runtime is commensurate with or more stable than MemP, with millisecond-level operations negligible relative to LLM generation.
  • Limitations: Step-wise updates can introduce high-variance noise in long-horizon trajectories, while low task similarity limits direct experience transfer.
  • Limitations: False-positive verifier feedback can produce reward hacking and rapidly propagate erroneous high-utility memories.

H. Case Study: High-Utility Failure Analysis (Near-Misses)

Near-miss case studies show how retrieved failure reflections encode transferable operational lessons. Their utilities and successful target-task retrievals illustrate how MEMRL prioritizes practically useful memories.

  • Case-study design: The case studies analyze high-value near-miss memories from a 10-epoch OS-interaction run.Each case includes an origin task, retrieved reflection, explanation, and target task.
  • Configuration editing: Handling commented configuration defaults explicitly generalizes across configuration-editing tasks.The reflection recommends uncomment-or-append behavior for naive sed replacements.
  • Utility outcomes: 100.0% utility was recorded for the selected memory in the first configuration-editing example.
  • Configuration editing: An idempotent editing pattern prevents duplicate appended lines by using a single-pass edit or checking for existing content.
  • Utility outcomes: 91.7% utility was recorded for the selected memory in the duplicate-line reflection example.
  • Verification: Explicit post-checks reduce uncertainty caused by silent command execution and validate exit status, files, permissions, and ownership.
  • Utility outcomes: 88.5% utility was recorded for the selected memory in the silent-execution reflection example.
  • Process management: Process-management reflections recommend capturing the PID once, verifying artifacts, and sequencing commands with error handling.
Loading 2601.03192v2…