Contextual Agentic Memory is a Memo, Not True Memory

Binyan Xu; Xilin Dai; Kehuan Zhang

Source-linked AI summary

Contextual Agentic Memory is a Memo, Not True Memory

Binyan Xu, Xilin Dai, Kehuan Zhang

arXiv:2604.27707v2Apr 30, 2026cs.AIcs.CL

TL;DR

The paper addresses the gap between retrieval-based persistence and durable learning in agentic systems. It formalizes the distinction using adaptive-state and context-based analyses, Complementary Learning Systems theory, and a conditional generalization argument. It concludes that scoped retrieval and separately governed adaptation should coexist when durable transfer justifies their costs and risks.

Problem
Current agentic memory systems often treat retrieval and external persistence as memory, despite not changing the generator’s context-independent adaptive state.
Method
The paper distinguishes changes to adaptive state θ from changes to context C, grounds the distinction in Complementary Learning Systems theory, and formalizes a conditional Generalization Gap.
Results
The paper concludes that retrieval-based systems can improve memory-assisted performance without developing context-independent competence, while a positive contextual error margin yields a conditional retrieval sample-complexity gap.
Takeaways & Limitations
The paper advocates pairing scoped episodic retrieval with a separately governed adaptation channel when durable transfer warrants its cost and risk.
Takeaways & Limitations
The performance claim is conditional, lacks a compute-matched frontier-model experiment, and does not demonstrate reliable large-scale consolidation; existing comparisons remain task-dependent.

Abstract

from arXiv · show

Current agentic memory systems (vector stores, retrieval-augmented generation, scratchpads, and context-window management) do not implement memory: they implement lookup. We argue that treating lookup as memory is a category error with provable consequences for agent capability, long-term learning, and security. Retrieval generalizes by similarity to stored cases; weight-based memory generalizes by applying abstract rules to inputs never seen before. Conflating the two produces agents that accumulate notes indefinitely without developing expertise, face a provable generalization ceiling on compositionally novel tasks that no increase in context size or retrieval quality can overcome, and are structurally vulnerable to persistent memory poisoning as injected content propagates across all future sessions. Drawing on Complementary Learning Systems theory from neuroscience, we show that biological intelligence solved this problem by pairing fast hippocampal exemplar storage with slow neocortical weight consolidation, and that current AI agents implement only the first half. We formalize these limitations, address four alternative views, and close with a co-existence proposal and a call to action for system builders, benchmark designers, and the memory community.

1 Introduction

The paper distinguishes retrieval-based persistence from durable learning, arguing that external memory can preserve artifacts without changing an agent’s adaptive state. It frames this distinction as consequential for generalization, evaluation, security, and system design.

1 Introduction: Modern agents commonly persist observations, critiques, skills, and other artifacts in external stores, treating retrieval as their default memory mechanism.Examples include MemGPT, Generative Agents, Reflexion, and Voyager.
1 Introduction: Retrieval is a practical, reversible, and auditable deployment choice, but it can be confused with persistent learning that changes how future inputs are processed.Retrieval supplies artifacts to a frozen generator, whereas durable updates change adaptive state.
1 Introduction: Complementary Learning Systems theory separates fast exemplar storage from slow rule-based consolidation, a distinction illustrated by novices using surface features while experts use structural principles.The paper argues that most deployed agents emphasize episodic storage while governed consolidation remains uncommon and immature.
1 Introduction: Researchers should distinguish memory-assisted improvement from context-independent competence, because recall-focused benchmarks can miss transfer gaps and persistent writes add attack surface.The paper calls for coordination among system builders, benchmark designers, and continual-learning researchers.
1 Introduction: The paper identifies four claims: external memory does not itself alter adaptive state, retrieval can face a conditional generalization gap, frozen competence remains unchanged, and persistent writes create security risk.It distinguishes memo-like acquired external state from internalized memory as durable adaptive change.

2 The Case: Agentic Memory Is a Memo, Not True Memory

Agentic memory systems primarily preserve external artifacts and inject retrieved content into context, rather than changing the generator’s persistent adaptive state. The paper contrasts this memo-like operation with weight-based internalization and positions durable adaptation as a separate substrate.

2 The Case: Agentic Memory Is a Memo, Not True Memory: Deployed agents commonly persist experience in external stores because reliable continual updating remains unsolved, but this substitute should not be evaluated as the learning mechanism itself.The paper describes the operational default as understandable yet conceptually distinct from durable learning.
2 The Case: Agentic Memory Is a Memo, Not True Memory: A state-based lens distinguishes changing adaptive state θ from changing context C through prompts, retrieval, tools, skills, or scratchpads.Changing C conditions generation on retrieved content, whereas changing θ modifies persistent weights, adapters, or learned policies.
2 The Case: Agentic Memory Is a Memo, Not True Memory: The experience-compression spectrum runs from raw traces to natural-language skills to parameterized rules, with increasing compression and generalizability.The appropriate substrate depends on compression: episodic traces fit external stores, skills can bridge context and weights, and durable rules require persistent adaptation.
2 The Case: Agentic Memory Is a Memo, Not True Memory: In the common write–retrieve loop, a past artifact is stored, a similar entry is retrieved, and the entry is injected into the current context without changing the frozen generator.The generator can reason over the artifact, but the experience remains contingent on supplying it.
2 The Case: Agentic Memory Is a Memo, Not True Memory: The diary contrast separates recall-dependent agentic memory from internalized learning that remains available across situations without consulting the original artifact.Reflexion, Voyager, and Generative Agents are presented as retaining external artifacts while using the same frozen model across sessions.
2 The Case: Agentic Memory Is a Memo, Not True Memory: ROME, MEMIT, and ParamMem establish weight-update primitives, but they do not constitute a solved frontier-scale consolidation system.The paper presents these methods as evidence that durable update mechanisms exist while leaving large-scale consolidation unresolved.

3 Evidence: Four Structural Limitations

The paper identifies structural, dynamic, and security limitations of treating retrieval as memory. Its formal analysis contrasts similarity-based lookup with parametric learning, while neuroscience and attack evidence motivate durable consolidation alongside scoped retrieval.

Definitional: Retrieval accesses stored cases by similarity, whereas internalized adaptation applies principles independently of the artifacts that supplied them.An external store alone cannot make a newly acquired rule persist independently of its stated or demonstrated artifact.
Structural: The paper frames the comparison through compositional sample complexity using the same experience budget for retrieval and fine-tuning.The framework evaluates generalization to unseen combinations of known concepts under transparent, testable assumptions.
Structural: Under Assumption 3.2, retrieval requires stored compositional examples, while an appropriate parametric learner can achieve CGC(MP, D) ≥1 −δ with probability at least 1 −β.The separation compares a coverage requirement for retrieval with hypothesis-class complexity d for parametric learning.
Structural: For structured operators with d = O(k), the retrieval-to-parametric sample-complexity ratio is Ω(k), and for d = O(1) it is Ω(k2).This conditional result requires a uniform positive contextual margin; if contextual accuracy reaches the target, the lower bound becomes non-binding.
Dynamic: Exclusively C-engineering leaves θ unchanged across sessions, so logged experience can improve assisted performance without changing context-independent composition competence.Every session uses the same frozen weights, while consolidation that changes the model itself would constitute a durable learning channel.
Dynamic: Cognitive evidence links expertise to structural reorganization through consolidation, whereas external accumulation leaves the generator at the novice stage.The paper connects hippocampal traces and neocortical weight changes to generalized representations rather than mere accumulation of examples.
Security: Persistent memory creates a cross-session injection path: MINJA reported 98.2% injection success, PoisonedRAG 90% attack success with five adversarial texts per query, and InjecAgent found memory-writing agents more vulnerable than stateless ones.The threat model is conditional: provenance checks, write gating, filtering, and deletion can interrupt the path.
Security: Security risks differ by reachability: external entries are inspectable and deletable, while consolidation introduces a separately gated update path with potentially larger blast radius and harder example-level unlearning.The paper therefore motivates scoped retrieval for raw experience and gated consolidation for durable adaptation rather than a universal safety ranking.

4 Alternative Views

The paper considers four objections to distinguishing retrieval from persistent adaptive learning. It argues that larger contexts, in-context rule induction, compression-spectrum views, and learned retrieval do not erase the distinction when persistence or held-out generalization remains limited.

Alternative 1: Context windows: Larger context windows can improve contextual accuracy, but when a positive held-out margin remains, they do not change persistent adaptive state.The theorem abstains if contextual accuracy reaches the target; otherwise, window growth alone does not remove the conditional coverage bound.
Alternative 2: In-context learning: In-context learning can induce task-specific operators, yet the induced update remains ephemeral when demonstrations or the induced rule are withheld.The performance claim applies only below the target contextual accuracy, while the persistence distinction applies regardless of contextual capability.
Alternative 3: Compression spectrum: A compression spectrum for memory, skills, and rules does not make their substrates interchangeable, because generalized rules or programs may remain dependent on retrieval.The paper treats compression level and persistence substrate as separate dimensions.
Alternative 4: Learned retrieval: Learned retrieval can improve contextual performance without updating the base generator, but an experience-updated retrieval policy is itself a persistent learning channel.The paper therefore places such policies in the adaptive row rather than treating them as ordinary fixed retrieval.

5 Call to Action

The paper proposes coexistence: retain retrieval for inspectable episodic artifacts while adding a separately governed consolidation pathway for durable adaptation. Builders should validate and secure that pathway, and benchmark designers should measure transfer and representation change rather than recall alone.

System builders: Agentic memory should remain episodic lookup for recent context, tool outputs, evidence, changing facts, and scoped state.These artifacts can be inspected, reproduced, filtered, or deleted without changing shared behavior.
System builders: Durable post-deployment adaptation requires an asynchronous pathway from episodic experience to persistent adaptive state.Possible mechanisms include fine-tuning, knowledge editing, test-time training, and self-distillation, but reliable frontier-scale continual updating remains unsolved.
System builders: Consolidation pipelines must track trace provenance and use controls suited to weight poisoning, whose effects may be harder to inspect or reverse than database poisoning.The two substrates require different controls rather than an assumed safety ordering.
System builders: Continual consolidation still faces unresolved choices about experience selection, timing, transfer validation, forgetting, recency bias, unlearning, reproducibility, certification, and compute.The paper frames these as joint mechanism and policy challenges.
Benchmark designers: Recall benchmarks can reward better-organized storage while providing limited evidence of context-independent transfer or genuine learning.The paper argues that near-perfect recall can coexist with zero genuine learning.
Benchmark designers: Benchmark designers should prioritize compositional generalization over time and distinguish representation change from improved access.The proposed evaluations expose isolated concepts during operation, then test unseen combinations and knowledge updates through weight- or representation-changing methods.
Benchmark designers: Longer context can improve assisted performance but does not by itself change persistent adaptive state, and raw scores across heterogeneous domains require calibration.The paper recommends prioritizing generalization over capacity.
Memory and continual learning: Agentic systems offer continual learning a natural experience stream, reward labels, and compositional novelty as a generalization criterion.Continual learning supplies consolidation machinery intended to convert episodic experience into durable knowledge while mitigating interference.

6 Related Work

Related work shows that deployed agent systems commonly use retrieval as their persistence mechanism, while existing studies support task-dependent complementarity among retrieval, in-context learning, and fine-tuning. Prior work also covers consolidation variants and memory-specific security attacks without establishing a universal winner.

Agentic memory: MemGPT, Generative Agents, Reflexion, and Voyager exemplify agent architectures that treat retrieval as the default persistence mechanism.The related work also notes newer memory systems and distinguishes this operational pattern from durable adaptive updating.
Comparative evidence: Empirical comparisons support task-dependent complementarity among retrieval, in-context learning, and fine-tuning rather than a universal winner.Directional evidence for parametric reflective memory exists, but not a universal compute-matched comparison.
Consolidation: RETRO and kNN-LM address static corpora, whereas newer work realizes variants of consolidation channels for accumulated experience.The paper separates static retrieval systems from post-deployment accumulation and consolidation.
Security: Security studies document injection attacks against memory-augmented agents, while the paper isolates a conditional persistent-write path rather than ranking substrate safety.This frames the security contribution as an attack-path distinction, not an absolute safety comparison.

7 Scope and Limitations

The paper’s performance separation is conditional rather than universal, and its empirical and engineering claims remain bounded by unresolved continual-learning and security challenges. The position would be falsified where fixed-state contextual systems consistently match updateable systems on held-out post-deployment compositions under equal resources.

Scope: The performance separation depends on measured contextual accuracy and an appropriate parametric learner for the target class.The paper does not claim a universal ranking of in-context learning and fine-tuning.
Evidence boundary: The paper provides no compute-matched frontier-model experiment demonstrating the Frozen Novice effect.Existing empirical comparisons are described as task-dependent.
Engineering limitations: Reliable large-scale consolidation remains un demonstrated because forgetting, recency bias, interference, unlearning, reproducibility, and certification remain substantial barriers.The security analysis compares attack paths rather than absolute safety.
Falsifiability: The performance position would be falsified if fixed-state contextual systems consistently matched updateable systems on held-out post-deployment compositions under the same experience and resource budget.In that domain, retrieval could be sufficient even though the systems update different state.

8 Conclusion

Agentic memory persists external state, but durable internalization requires a separately governed adaptation channel. The paper therefore recommends scoped episodic retrieval alongside persistent learning when durable transfer justifies its costs and risks.

Conclusion: External persistence can support sophisticated reasoning, but acquired capability remains contingent on supplying the stored state.The distinction is between assisted performance and context-independent competence.
Conclusion: The paper advocates pairing scoped episodic retrieval with a separately governed adaptation channel when durable transfer warrants its cost and risk.This preserves the usefulness of retrieval while separating it from durable learning.
Conclusion: These systems are useful filing systems, but durable internalization remains a different operation.The conclusion separates practical utility from persistent learned competence.
Conclusion: A stylized bounded-context example shows that retrieval may fail when more task-specific relations are required than can fit into context.For m > K, at least one necessary relation is absent under the stated assumptions.
Conclusion: The context-capacity example is a boundary case rather than a universal ordering because larger windows, compression, iterative retrieval, or supplied rules can remove the gap.It does not establish that fine-tuning will learn the required structure.

B Proof of Theorem 1 (Compositional Sample Complexity Separation)

The theorem compares retrieval with parametric learning on compositional tasks under a measurable contextual-accuracy premise. Retrieval faces coverage-dependent sample requirements, whereas parametric learning depends on the composition class's intrinsic complexity.

Setup: The theorem analyzes retrieval and parametric systems given the same labeled compositional training data and concept-pair task space.The retrieval system uses a frozen model with top-K retrieval, while the parametric system learns the composition operator.
Part 1: Retrieval lower bound: For stored concept pairs retrieval is correct, while novel pairs are limited to contextual accuracy at most ᾱ under the theorem’s assumption.The bound includes the frozen model’s in-context reasoning over retrieved examples.
Part 2: Parametric upper bound: i.i.d. examples allow an appropriate parametric learner to achieve error at most δ with probability at least 1 − β when the operator belongs to the hypothesis class.This yields CGC(MP, D) ≥ 1 − δ.
Part 2: Parametric upper bound: The parametric sample requirement depends on d and δ, not on k, the number of base concepts.This contrasts with retrieval’s coverage over the space of concept pairs.
Sample-complexity comparison: For structured operators with d = O(k), nR/nP = Ω(k), while operators with d = O(1) yield nR/nP = Ω(k^2).These ratios hold for fixed δ, β, and a uniform positive contextual margin.
Part 4: Dependence on contextual inference: The separation is conditional: if contextual inference reaches target accuracy, the theorem makes no retrieval lower-bound claim.Increasing K or improving retrieval can raise ᾱ and invalidate the premise when the target is reached.

C Proof of Proposition 1 (Information-Theoretic Bound on ICL Accuracy)

The proposition studies how much information retrieved labels provide about an unknown composition operator. Under an unstructured operator family, unseen outputs remain unpredictable, while stronger uniform margins require additional separation assumptions.

Part 1: Unstructured operators: The analysis considers unstructured operators in which outputs on distinct concept pairs are independent and uniformly distributed.Observing labels for some pairs therefore gives no information about the remaining pairs.
Part 1: Unstructured operators: For unseen pairs, Bayes-optimal prediction accuracy is exactly 1/|Y|, so ᾱ = 1/|Y|.This supplies an explicit contextual-accuracy bound for the unstructured case.
Part 2: Structured operators: The structured analysis bounds information about an operator index V from K observed labeled pairs by I(V ; Z) ≤ K log |Y|.Fano’s inequality is then used to derive an identification bound.
Part 2: Structured operators: A positive uniform prediction margin requires an explicit separation condition in prediction space.Without that separation, distinct posterior hypotheses may disagree on arbitrarily little probability mass.
Corrected Fano reduction: The corrected argument limits the auxiliary uniform-margin justification to cases where both the packing radius and the Fano term remain positive.Otherwise the contextual-accuracy premise must be measured empirically, without altering Theorem 1’s explicit assumption.

D Comparison of Continual Learning Methods

The paper presents a modular-arithmetic construction as a formal existence proof for a quadratic retrieval disadvantage and situates continual-learning methods as potential consolidation tools. The construction is intended to represent structured domains beyond modular arithmetic.

Comparison of Continual Learning Methods: Table 2 identifies representative continual-learning methods suitable for an offline consolidation pipeline, all enabling rule-based weight updates from agent experience.The table frames these methods as potential tools for persistent adaptation.
E.1 Modular arithmetic construction: The modular-arithmetic family uses k base concepts from Zp and an unknown domain-specific constant c in its composition operator.The hypothesis class consists of functions parameterized by c.
E.1 Modular arithmetic construction: A single labeled example identifies c, giving the parametric hypothesis class VC dimension d = 1.This makes the operator learnable from a constant number of informative examples.
E.1 Modular arithmetic construction: The parametric system requires nP = O(1/δ) examples, whereas retrieval requires nR = Ω(k^2) stored compositions for large p.The resulting separation ratio is nR/nP = Ω(k^2).
Naturalistic instantiations: The construction is a formal existence proof for domains with many base concepts and structured composition rules.The paper names clinical, legal, and engineering compositions as naturalistic instantiations.

E.2 Composition-error corollary

The corollary conditions its separation claim on the complete frozen contextual system retaining a positive held-out error margin. When that margin is absent, the proposition abstains and contextual memory is sufficient for the target.

εC = 1 −α(K) measures the complete frozen contextual system’s held-out error on compositions, including retrieval, prompting, and rule induction.
If εC > δ and an adapted learner meets Theorem 1’s realizability and sample conditions, the conditional error-margin proposition applies with probability at least 1 −β.
If εC ≤δ, the proposition abstains and makes no separation claim because contextual memory is sufficient for the target.