PaperlayerView exact revision on arXiv

Source-linked AI summary

Memento 2: Learning by Stateful Reflective Memory

Jun Wang

arXiv:2512.22716v3cs.AIcs.CVcs.LG

TL;DR

The paper addresses the sample demands of gradient-based learning and task adaptation by formalizing continual learning through episodic memory and reflection in LLM agents. It introduces a read–write reflective decision framework and reports that, as memory covers the state space more fully, the resulting policy converges to the optimal solution.

  • Problem

    Gradient-based learning typically requires many examples, and adapting pretrained LLMs to specific tasks still demands additional curated data or human annotations.

    Source: §1 · p. 2

  • Method

    The paper formalizes reflection as a read–write process over episodic memory and develops a framework integrating retrieval with KL-regularised soft policy iteration.

    Source: §2 · p. 4, §6 · p. 21

  • Results

    As episodic memory grows and sufficiently covers the state space, both LLM approximation and memory coverage errors vanish, and the composite policy and value function converge to the underlying MDP’s optimal solution.

    Source: §4.3 · p. 19

  • Takeaways & Limitations

    The results establish that iterative read–write reflection over expanding memory is sufficient to recover optimal performance in the limit without parameter updates.

    Source: §1 · p. 4, §4.3 · p. 19

    The framework may face computational challenges as memory grows, and the impact of embedding quality on performance is not fully addressed.

    Source: §6 · p. 21

Abstract

from arXiv · show

We present a theoretical study of continual and experiential learning in large language model agents that combine episodic memory with reinforcement learning. We argue that the key mechanism for continual adaptation, without updating model parameters, is reflection: the agent's ability to use past experience to guide future actions. Empirical findings suggest that episodic, experience-driven reflection enables generalised adaptation across a wide range of open-ended, long-horizon tasks. This indicates that efficient learning can occur during deployment and weakens the traditional separation between training and testing. Motivated by this, we introduce the Stateful Reflective Decision Process, a formal model of reflective memory dynamics. In this abstraction, an agent maintains an episodic memory and performs two core operations. Writing stores interaction outcomes and plays the role of policy evaluation. Reading retrieves relevant past cases to inform decisions and plays the role of policy improvement. This perspective treats reflective memory as a control object that can be analysed using classical reinforcement learning tools. We then develop a read-write reflective learning framework by integrating retrieval into soft policy iteration and establish convergence guarantees. We show that as memory grows and provides denser coverage of the state space, the resulting composite policy converges to the optimal solution. Overall, this framework connects practical memory-based methods with principled reinforcement learning, providing a rigorous mathematical basis for building reflective, memory-embedded agents capable of continual general-purpose learning.

1 Introduction

The introduction motivates learning from experience without repeated parameter updates, and frames reflection over episodic memory as a route to continual adaptation. It presents the Stateful Reflective Decision Process as a formal account of memory-based learning in which reading improves decisions and writing stores outcomes.

1 Introduction

  • The introduction identifies high sample complexity in gradient-based learning and the additional curated data or annotations required to adapt pretrained LLMs to specific tasks.It contrasts this with human use of semantic representations, episodic memory, and reflective reuse of experience.Source: §1 · p. 2
  • The paper describes three reflection modes: prompt exemplars, environmental feedback after outputs, and internal reasoning or planning before answers.These approaches share intermediate information gathering or computation intended to support adaptation beyond training data.Source: §1 · pp. 2–3
  • Earlier work introduced memory-based LLM agents and reported continual self-improvement across software testing, automated data science, and deep research without model retraining.The paper builds its theoretical analysis on this earlier empirical line.Source: §1 · p. 3
  • The paper proposes learning through reflection over retrieved episodic experience, rather than parameter updates, and aims to unify internal reasoning, environmental interaction, and human-labelled data.The framework is motivated as a sample-efficient alternative to data-intensive training.Source: §1 · p. 3
  • The Stateful Reflective Decision Process incorporates episodic memory into the agent’s state and models decisions as recall followed by a response, with experience written back over time.The resulting enlarged state preserves a Markov formulation while memory evolves.Source: §1 · pp. 3–4
  • Reading retrieved cases supports policy improvement, while writing interaction outcomes to memory serves as policy evaluation.The paper relates these operations to episodic control and reinforcement learning.Source: §1 · p. 4

2 Stateful Reflection as Generalised Learning Methods

This section defines reflection as a read–write learning loop over episodic memory and sets out how that loop connects to policy iteration. It distinguishes the proposed stateful, experience-updated memory from isolated case retrieval or static external documents.

2 Stateful Reflection as Generalised Learning Methods

  • Reflection is defined as an iterative process in which an LLM agent improves its effective policy through coordinated memory, reasoning, and environmental feedback, using a read step followed by a write step.The section presents it as a closed learning loop rather than a single operation.Source: §2 · p. 4
  • Episodic memory is a finite, growing collection whose items record a state, action, reward, and subsequent state, with a write operation updating the collection.These stored interactions provide the cases available to later reads.Source: §2 · p. 5
  • The workflow reads relevant episodic items or summaries for a query, generates and executes an action, receives evaluative feedback, and writes the resulting episode to memory.The section defines episodic memory as a structured repository of past interaction episodes.Source: §2 · pp. 4–5
  • The section casts reading as policy improvement and writing interaction-driven feedback as policy evaluation, making alternating reflection analogous to policy iteration.It states convergence to an optimal or self-consistent policy under assumptions that reading does not reduce policy quality and writing provides informative evaluation signals.Source: §2 · p. 5
  • SRDP models memory as dynamic and stateful, distinguishing it from static-document RAG and from case-based reasoning that treats retrieved cases as isolated problems.Its sequential decision framing models temporal dependencies, memory evolution, and long-horizon effects; the section also cites empirical learning without fine-tuning across data-science, software-testing, and deep-research agents.Source: §2 · pp. 5–6

3 A Model for Reflective Process over Episodic Experience

The section extends the MDP framework to model LLM decisions informed by episodic memory. It formalizes retrieval and memory evolution in an augmented state space where the process is Markovian.

3.1 Preliminary: Markov Decision Process

  • An MDP models decisions with states, actions, transitions, rewards, and a discount factor; for LLM agents, states can be prompts or reasoning steps and actions are generated outputs.The LLM policy maps states or prompts to action distributions, with its parameters frozen in this formulation.Source: §3.1 · pp. 6–7

3.2 Stateful Reflective Decision Process

  • SRDP extends the MDP with episodic memory, a retrieval policy, and an LLM kernel that generates actions conditioned on the state and retrieved case.Its two-stage decision first retrieves a memory case, then generates an environment action conditioned on that case.Source: §3.2 · p. 7, §3.2.1 · p. 8

3.2.1 Definition

  • The composite SRDP policy combines retrieval from episodic memory with LLM-generated actions, spanning memory-based control and decisions relying on the LLM when relevant experience is absent.The memory grows as experience tuples are written, expanding the agent’s retrieval options.Source: §3.2.1 · pp. 8–9

3.2.2 Transforming to Reflected MDP

  • The Reflected MDP augments the state to include environment state and episodic memory, restoring the Markov property and enabling a classical control formulation.The augmented state is x = (s, M), and retrieval is the controllable policy while LLM-generated actions are absorbed into the transition dynamics.Source: §3.2.2 · pp. 9–10
  • In the Reflected MDP, retrieval and LLM action generation induce transitions and rewards over the augmented state, while writing appends new experience to memory.The LLM is treated as fixed or slow-evolving, leaving retrieval as the agent’s primary policy.Source: §3.2.2 · pp. 9–10

4 Read–Write Reflective Learning

This section formalizes read–write reflection as policy iteration over episodic memory, extending the framework to slower online memory updates. It examines how LLM approximation and retrieval errors relate to reflective policy error and how increasing memory coverage can support asymptotically optimal behavior without parameter updates.

4.1 Policy Iteration

  • Parzen retrieval mixes similar stored cases with a void case for internal-knowledge action, while KL-regularized policy improvement preserves reward feedback and discovery.The void case receives nonzero prior probability, allowing action even when memory has no relevant cases.Source: §4.1 · pp. 11–12
  • Under bounded-reward and stationary-memory assumptions, Parzen-KL soft policy iteration converges to a fixed point optimal for its KL-regularized objective.For a fixed policy, the KL-regularized evaluation operator is a contraction with a unique fixed point.Source: §4.1 · p. 12

4.2 Two-Time-Scale Convergence with Online Memory Rewriting

  • With memory updated more slowly than policy and value estimates, the fast variables track stationary-memory fixed points and the joint process converges to a limiting set.The analysis assumes the two-time-scale conditions, including bounded iterates and a compact memory attractor; slower updates support stability and tracking as memory evolves.Source: §4.2 · pp. 13–15

4.3 The Role of Reflection and Memory towards Optimal Policy

  • The algorithm couples fast KL-greedy reading and policy evaluation with slow memory writing, yielding monotonic fast-scale policy improvement and convergence to the reflected-MDP optimum.Each memory update changes the effective reflected MDP and its policy-evaluation problem.Source: §4.3 · p. 16
  • For fixed memory the cycle converges to a locally optimal policy, while expanding coverage can drive the retrieval-plus-LLM composite policy toward the true optimum under local consistency.The LLM local-consistency condition requires its action distribution to approach the optimal policy as retrieved cases get nearer to the state.Source: §4.3 · p. 17
  • A small εLLM(r) lets the LLM infer near-optimal actions from approximate local context, enabling larger memory neighborhoods and fewer cases for efficient convergence.The error function measures the LLM’s effective radius of competence: how far local contextual cues can support near-optimal behavior.Source: §4.3 · pp. 17–18
  • Theorem 14 bounds the value difference between the optimal and composite policies by a term proportional to their maximum total-variation policy discrepancy.The bound scales with the discrepancy ΔM and the MDP reward and discount parameters.Source: §4.3 · p. 18
  • As episodic memory grows and covers the state space, the policy discrepancy and value gap vanish, so the composite policy converges to the optimal solution.The discrepancy decomposes into LLM approximation error and memory coverage error; both vanish when locally relevant cases enable near-optimal action approximation.Source: §4.3 · p. 19
  • The composite policy’s total-variation discrepancy is bounded by the LLM approximation error εLLM(rM) plus retrieval error δM.Here, δM(s) is the probability of retrieving a case outside radius rM(s) despite a case existing within that radius.Source: §4.3 · pp. 18–19

5 Related Work

The related work distinguishes reflective memory from implicit neural memory, direct episodic control, and case-based retrieval, and situates the framework’s retrieval regularization and convergence analysis within reinforcement learning.

5 Related Work

  • HAMI shares the biological motivation, while this framework instead uses probabilistic Parzen-KL retrieval and information-theoretic regularization.The paper presents these choices as underpinning stability and scalability.Source: §5 · p. 20
  • Unlike implicit recurrent memory, reflective memory lets the LLM reason from retrieved context rather than encoding history in a hidden state.The paper notes that hidden-state memory makes long-horizon retention, interpretability, and revision difficult.Source: §5 · p. 20
  • Unlike episodic control methods [42], which reuse actions or approximate values through local interpolation, this method uses memory as a substrate for reflective generalisation.Density-based similarity priors and KL-regularized updates support its retrieval mechanism.Source: §5 · p. 20
  • Prior work on CBR-based LLMs [23] [24] provides empirical evidence, particularly in data science and coding tasks, that LLMs effectively revise retrieved cases, enabling strong generalisation to new tasks.Within the framework, CBR-based LLMs are viewed as a stateless special case.Source: §5 · p. 20
  • Unlike experience replay [44], which samples transitions uniformly or by temporal-difference error, this approach uses density-aware retrieval for context-sensitive access.The paper argues that replay lacks the structured similarity-based retrieval needed for generalization across diverse contexts.Source: §5 · p. 20
  • KL regularization applies to both policy and retrieval, balancing high-value memory use with exploration while maintaining consistency with the similarity structure.The convergence analysis adapts entropy-regularized MDP results and stochastic approximation to memory-based retrieval and simultaneous value and memory updates.Source: §5 · pp. 20–21

6 Conclusions and Future Work

The conclusion formalizes memory-based reflection as an MDP extension and gives convergence guarantees for read-write learning. It identifies practical design considerations and proposes internal memory architectures as a direction for future work.

6 Conclusions and Future Work

  • SRDP augments an MDP with episodic memory, where retrieval improves the policy and action generation uses retrieved cases; the analysis proves convergence and asymptotic optimality as memory covers the state space.It also establishes two-timescale convergence when memory evolves more slowly than policy updates.Source: §6 · p. 21
  • Effective memory-augmented agents require density-aware retrieval, soft entropy-regularized improvements, systematic memory growth, and tuning of the void case c0.Similarity metrics and bandwidths should adapt as memory evolves to maintain efficiency and relevance.Source: §6 · p. 21
  • The framework may face computational challenges as memory grows, and the effect of embedding quality on performance remains unaddressed.Source: §6 · p. 21

6.1 Internalising Episodic Memory into the LLM Architecture

  • A proposed next step is to internalize episodic memory and read-write reflection in the LLM, replacing external retrieval with an online-updated internal state.The proposed state retains the SRDP interpretation of reading as policy improvement and writing as policy evaluation.Source: §6.1 · pp. 21–22, §6 · p. 21
  • A hybrid architecture could keep the LLM fixed or update it slowly while updating an internal memory module or retrieval head online, separating fast policy iteration from slower memory evolution.The paper proposes this two-timescale design as a way to retain the reflected-MDP stability analysis while pursuing efficiency and scalability.Source: §6.1 · p. 22
  • The paper proposes viewing SRDP as nested optimization, with fixed representations and long-term objectives outside online read-write adaptation.This could help characterize when reflection suffices, when parameter adaptation is needed, and how to combine them without destabilizing long-horizon control.Source: §6.1 · p. 22
  • Memory consolidation could implement writing by compressing episodic experience into a latent state, with slow consolidation and fast querying.The paper also identifies memory coverage, approximation error, and guarantees for neural internal memories as open theoretical questions.Source: §6.1 · p. 22

A Proof of Lemma 5

The proof derives equation (11) from the Lagrangian for (10), using a partial derivative with respect to ν(c).

A Proof of Lemma 5

B Proof of Theorem 8

The proof extends contraction analysis for KL-regularized evaluation to memory-dependent Parzen base measures. Its supporting log-sum-exp bounds control changes in values and base measures.

B Proof of Theorem 8

  • State-dependent base measures vary with memory, so the proof must establish contraction for the induced log-sum-exp operators.Lemma 16 extends a log-sum-exp Lipschitz bound to Parzen base measures and memory updates.Source: B Proof of Theorem 8 · p. 27, §B.1 · pp. 27–28

B.1 Bi-Lipschitz Properties

  • Under bounded Q-values, the log-sum-exp function is Lipschitz in the base measure, with its change bounded by 2αe^(∆Q/α) times the total variation distance.Source: §B.1 · pp. 27–28
  • Under the same boundedness condition, the joint change is bounded by the Q-vector sup-norm difference plus αe^(∆Q/α) times the base-measure L1 difference.Source: §B.1 · p. 28

B.2 Contraction of the KL-Regularised Evaluation Operator

This section establishes contraction and fixed-point properties for KL-regularised evaluation, then presents monotone-improvement and convergence results for read-write iterations.

B.2 Contraction of the KL-Regularised Evaluation Operator

  • The KL-regularised evaluation operator is a γ-contraction, so it has a unique fixed point by Banach’s fixed-point theorem.The theorem fixes a memory state and its induced base measure.Source: §B.2 · p. 29
  • Under bounded Q, the evaluation operator is stable to changes in memory-induced base measures, and this perturbation bound supports later fixed-point continuity and tracking results.The passages identify its use in deriving the moving fixed point’s Lipschitz continuity and a two-timescale tracking recursion.Source: §B.2 · p. 29

B.3 Monotone Improvement of Read-Write Iterations

  • Theorem 19 establishes monotone improvement under Parzen-KL for the read-write iterations.The supplied passage names the result but does not specify its inequality.Source: §B.3 · p. 30

B.4 Convergence of Read-Write Iterations

  • Bounded, monotonically increasing soft values converge, and continuity of the improvement mapping makes the entire sequence converge to a uniquely determined fixed point.The proof also uses convergence of each evaluation stage to its unique Q fixed point.Source: §B.4 · pp. 30–31

C Lipschitz Properties of Memory Updates

This section treats memory through its induced base measure and establishes Lipschitz properties that relate memory changes to learning targets and updates.

Practical memory update patterns

  • Experience replay, sliding-window replacement, and importance-weight updates are listed as practical memory-update patterns.The supplied passages name the patterns but do not specify the first pattern’s update mechanism.Source: C Lipschitz Properties of Memory Updates · p. 31

Memory as an operator-valued state

  • The algorithm accesses memory through the base measure used by the Parzen-KL prior and KL-regularised evaluation operator, rather than as a raw set of cases.The memory distance is defined at the operator level through the induced base measures.Source: C Lipschitz Properties of Memory Updates · p. 31

Lipschitz continuity of Parzen-KL targets

  • Under the bounded-Q assumption, the Parzen-KL targets are Lipschitz in memory, with bounds for both the value target and policy target.The proof combines the two target bounds and applies the memory-distance definition.Source: C Lipschitz Properties of Memory Updates · pp. 31–32

Lipschitz continuity of memory updates

  • For sample-form updates (14)–(15), the mean-field drift is Lipschitz in memory, with a constant scaling as O(e^∆Q/α).Memory enters the TD and policy targets through the log-sum-exp and policy mappings, respectively.Source: C Lipschitz Properties of Memory Updates · pp. 32–33

D Proof of Theorem 10

The proof analyzes fixed-memory learning as fast dynamics coupled to slower memory changes, establishing stability, memory sensitivity, and tracking conditions.

D.1 Fast Timescale: Fixed-Memory Dynamics

  • For fixed memory, KL-regularised policy evaluation and Parzen-KL policy improvement form the fast dynamics, whose ODE has a unique globally asymptotically stable equilibrium.The stability conclusion follows from Theorems 17 and 19.Source: §D.1 · p. 33

D.2 Perturbed Contraction and Fixed-Point Lipschitz Continuity

  • Under the stated contraction and memory-Lipschitz assumptions, the fixed point varies Lipschitzly with memory, with bound proportional to LF/(1−κ).The result is stated for the fixed point in Lemma 25.Source: §D.2 · p. 33, Then the fixed point satisfies · p. 33

D.3 Slow Timescale: Memory Dynamics

  • The slow memory ODE admits a compact attractor set, and the assumed practical update schemes change the induced base measure by order ρt.The listed schemes are experience replay, sliding windows, and importance-weighted memory.Source: §D.3 · p. 34

D.4 Tracking Lemma for Two-Timescale SA

  • The tracking lemma assumes a globally asymptotically stable fast equilibrium for each fixed memory state.The lemma relies on standard two-timescale stochastic approximation results.Source: §D.4 · p. 34

D.5 Derivation of the Tracking Recursion

The tracking argument compares fast variables with their memory-dependent equilibrium and uses the separation between update timescales to control drift.

D.5 Derivation of the Tracking Recursion

  • The tracking error is defined for the fast update relative to the memory-dependent equilibrium.The supplied passages introduce the error and the fast update form without showing their full expressions.Source: §D.5 · p. 34
  • With λ = 1 − κ > 0 and L denoting the constant from Lemma 25, ∥Mₜ₊₁ − Mₜ∥ = O(ρₜ) and ρₜ = o(ηₜ), making the drift negligible relative to contraction.Source: §D.5 · p. 34
  • Because memory changes by O(ρt) while ρt = o(ηt), its drift is negligible relative to the fast contraction.This timescale separation is stated directly in the tracking argument.Source: §D.5 · p. 34

D.6 Convergence to the Limit Set

The convergence proof reports almost-sure convergence of the error term and memory, then locates limit points of the process. The accompanying notation explains how augmenting the state with memory makes the agent–memory interaction amenable to classical reinforcement-learning analysis.

D.6 Convergence to the Limit Set

  • D.6 reports ε_t→0 almost surely and M_t→M∞, and uses these facts to locate limit points of (Z_t, M_t).The passage says the proof is completed but does not include the full description of the limit set.Source: §D.6 · p. 35

E Notations of MDP, SRDP, and Reflected MDP

  • The Reflected MDP augments the state with memory and absorbs fixed LLM behaviour into effective transition and reward kernels, restoring Markovianity.This permits classical reinforcement-learning analysis while preserving the agent–memory–LLM interaction dynamics.Source: E Notations of MDP, SRDP, and… · p. 35
  • Unlike the underlying MDP, the SRDP models episodic memory and a two-stage decision in which retrieval selects a case that conditions LLM action generation.The underlying MDP models only environment dynamics and abstracts away the agent’s internal structure.Source: E Notations of MDP, SRDP, and… · p. 35

Related papers

Loading 2512.22716v3…