Source-linked AI summary

AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents

Jiafeng Liang, Hao Li, Chang Li, Jiaqi Zhou, Shixin Jiang, Zekun Wang, Changkai Ji, Zhihao Zhu, Runxuan Liu, Tao Ren, Jinlan Fu, See-Kiong Ng, Xia Liang, Ming Liu, Bing Qin

arXiv:2512.23343v1cs.CLcs.AIcs.CV

TL;DR

Existing work struggles to integrate human memory mechanisms into agent memory, while stateless LLMs cannot maintain cross-session continuity or accumulate experience. This survey connects cognitive neuroscience and LLM-driven agents, reviewing their memory systems and finding that multimodal memory improves performance across varied environments.

  • Problem

    Interdisciplinary barriers leave existing work struggling to assimilate human memory mechanisms, while stateless LLMs lack cross-session continuity and accumulated experience.

    Source: §1 · p. 2
  • Method

    The survey integrates cognitive neuroscience and agent research to compare memory definitions, taxonomies, storage, management, benchmarks, security, and future directions.

    Source: §1 · p. 2, §10 · p. 33
  • Results

    Existing agent systems show that multimodal memory consistently improves performance across a wide range of environments.

    Source: §9.1 · p. 32
  • Takeaways & Limitations

    Episodic-oriented benchmarks assess memory-driven problem solving and robustness of state tracking under real-environment noise.

    Source: §7.2 · p. 30

    Seamless memory representations and operations across modalities that preserve semantic consistency and temporal alignment remain unsolved.

    Source: §9.1 · p. 32

Abstract

from arXiv · show

Memory serves as the pivotal nexus bridging past and future, providing both humans and AI systems with invaluable concepts and experience to navigate complex tasks. Recent research on autonomous agents has increasingly focused on designing efficient memory workflows by drawing on cognitive neuroscience. However, constrained by interdisciplinary barriers, existing works struggle to assimilate the essence of human memory mechanisms. To bridge this gap, we systematically synthesizes interdisciplinary knowledge of memory, connecting insights from cognitive neuroscience with LLM-driven agents. Specifically, we first elucidate the definition and function of memory along a progressive trajectory from cognitive neuroscience through LLMs to agents. We then provide a comparative analysis of memory taxonomy, storage mechanisms, and the complete management lifecycle from both biological and artificial perspectives. Subsequently, we review the mainstream benchmarks for evaluating agent memory. Additionally, we explore memory security from dual perspectives of attack and defense. Finally, we envision future research directions, with a focus on multimodal memory systems and skill acquisition.

1 INTRODUCTION

The introduction frames memory as the link between past experience and future decisions, and argues that agent memory research needs deeper integration with cognitive neuroscience. The survey connects biological and artificial memory across definitions, taxonomies, storage, management, evaluation, security, and future directions.

1 INTRODUCTION

  • The survey integrates cognitive-neuroscience insights with LLM-driven agents to address the limited biological depth and disciplinary separation of prior work.It presents memory progressively from the human brain through LLMs to agents.Source: §1 · p. 2
  • It compares biological and agent memory taxonomies and storage, distinguishing procedural experience from conceptual knowledge and within-trajectory from cross-trajectory persistence.The survey also compares biological and artificial storage mechanisms by location and format.Source: §1 · p. 2
  • The survey examines the complete memory-management lifecycle, from extraction and updating through retrieval and utilization in brains and agents.It describes hippocampal-neocortical stabilization and replay alongside agent record distillation, refreshing, retrieval, and reasoning use.Source: §1 · p. 2
  • It categorizes agent-memory benchmarks as semantic-oriented or episodic-oriented and reviews memory security from both attack and defense perspectives.The attack examples include sensitive-information extraction, backdoors, and judgment degradation; defenses include retrieval purification and harmful-response blocking.Source: §1 · p. 3
  • The survey identifies multimodal memory and transferable agent skills as future research directions.It describes skills as composable, reusable, and portable resources shared across heterogeneous agents.Source: §1 · p. 3

2 MEMORY DEFINITIONS

The paper traces memory from human information processing through LLM storage forms to agent memory architectures and their management. It contrasts the substrates and limits of these systems, then surveys how agents store, schedule, retrieve, and evolve memories.

2.1 Memory from the Perspective of Cognitive Neuroscience

  • Human memory encodes and integrates new information, then consolidates or retrieves stored representations to guide present cognition and future behavior.The exposition proceeds from neuroscience to LLMs and then agents.Source: §2.1 · p. 3, §2 · p. 3

2.2 Memory from the Perspective of Large Language Models

  • LLM memory comprises parametric, working, and explicit external forms, corresponding respectively to model weights, context windows, and external storage and retrieval.Parametric knowledge is internalized during training, but frozen parameters can lag new facts and probabilistic representations can hallucinate; high training costs and forgetting risk also constrain updates.Source: §2.2 · p. 3, §2.2.1 · p. 4, §2.2.2 · p. 4
  • Working memory uses the context window for real-time interaction and reasoning, but long sequences are costly and positional bias makes it unsuitable as a stable long-term knowledge base.The model tends to attend more to the beginning and end of the window than to its middle.Source: §2.2.2 · p. 4
  • Explicit external memory separates storage from model weights, enabling updated knowledge and provenance tracking while retrieval latency, irrelevant-context noise, and index complexity remain bottlenecks.Retrieval-augmented generation is a prevailing paradigm using vector databases or knowledge graphs.Source: §2.2.3 · p. 4

2.3 Memory from the Perspective of Agents

  • Agent memory is a dynamic cognitive architecture for identity persistence, experience accumulation, and long-horizon planning, rather than merely a storage medium.The paper organizes its account around structured storage, dynamic scheduling, and cognitive processing and evolution.Source: §2.3 · p. 5

2.3.1 Structured Storage.

  • Agent storage ranges from timestamped memory objects and recursively summarized memory trees to skill libraries that map semantic descriptions to executable code.These formats turn interaction records and exploration experiences into indexable or reusable structures.Source: §2.3.1 · p. 5

2.3.2 Dynamic Scheduling Mechanisms.

  • Scheduling mechanisms route and update information under attention limits, using compression, time-slicing, caching, or learned policies adapted to real-time states.Dynamic scheduling is described as addressing the tension between limited attention and large memory stores.Source: §2.3.2 · pp. 5–6

2.3.3 Cognitive Processing and Evolution.

  • Agents reflect on past behavior, abstract reusable knowledge across domains, and manage forgetting or memory-pool growth; unlike static RAG, agent memory accumulates interaction and environmental feedback.Examples include extracting lessons from successful and unsuccessful experiences, removing low-value information, and maintaining a fixed-size latent memory pool.Source: §2.3.3 · p. 6, §2.3.4 · p. 6

3 MEMORY UTILITY

Memory extends agent capabilities through context management, personalization, and experience-based reasoning. To address long-horizon context limits, the paper contrasts heuristic organization with autonomous memory optimization.

3 MEMORY UTILITY

  • Memory supports agents by alleviating context-window constraints, enabling long-term personalization, and driving experience-based reasoning through reflection and planning.These three utilities are presented as core functions of memory in agent architectures.Source: §3 · p. 7, Figure 1 · p. 7

3.1 Breaking Context Window Constraints

  • Long-horizon interactions remain constrained by attention's quadratic complexity and lost-in-the-middle effects, so memory reconstructs context within limited attention budgets rather than merely truncating it.The paper distinguishes hierarchical compression and virtualization indexing from autonomous memory optimization.Source: §3.1 · pp. 7–8

3.1.1 Heuristic Context Design.

  • Heuristic context designs preserve global threads while hiding local details through virtual paging, gist indexes, task-structured folding, or iterative summarization.These strategies organize what remains in the active context and retrieve or preserve details as needed.Source: §3.1.1 · p. 8

3.1 Breaking Context Window Constraints

  • Autonomous memory optimization makes management learnable: agents can optimize summaries, treat summarization and deletion as actions, or fold context at task-relevant levels of detail.Context folding can toggle between granular condensation and deeper consolidation, helping reduce loss of critical details in long-horizon web interactions.Source: §3.1.2 · p. 8

3.2 Constructing Long-term Personalized Profiles

Memory complements fixed parametric knowledge by turning interaction history into evolving user profiles and using those profiles to guide personalized decisions. The section covers profile construction and preference-aligned execution.

3.2 Constructing Long-term Personalized Profiles

  • Memory transforms fragmented interactions into evolving user profiles that support adaptation to individual preferences in long-term interactions.The section divides this utility into profile construction and preference-aligned execution.Source: §3.2 · pp. 8–9

3.2.1 Profile Construction.

  • Agents can review interaction streams to infer user traits and motivations, retrieve relevant preferences, and maintain structured attributes and relationship states across long interactions.These mechanisms aim to sustain a coherent representation of who the user is and how the relationship stands.Source: §3.2.1 · p. 9

3.2 Constructing Long-term Personalized Profiles

  • Retrieved interaction patterns and memory banks can guide web-task decisions according to user habits, including test-time personalization without parameter updates.Memory can act as a supervisory signal or dynamic constraint for preference-aligned execution.Source: §3.2.2 · p. 9

3.3 Driving Experience-based Reasoning

Memory supports experience-based reasoning in two ways: it retrieves experience to guide plans and converts successful processes into reusable skills. The section contrasts strategic guidance with procedural solidification.

3.3 Driving Experience-based Reasoning

  • Memory can also solidify frequent successful paths into workflows, code or templates, allowing agents to invoke procedures instead of repeating planning.The two paradigms are described as experience retrieval for better plans and procedural reuse for efficient task completion.Source: §3.3 · p. 9, §3.3.2 · p. 10

3.3.1 Strategic Guidance.

  • Strategic guidance uses relevant experiences, guidelines or internalized intuition to steer action selection and improve reasoning.Methods include retrieving past trajectories or guidelines and summarizing successful and failed trajectories to guide inference [391] [159] [32].Source: §3.3.1 · pp. 9–10, §3.3 · p. 9

3.3 Driving Experience-based Reasoning

  • Procedural solidification turns successful reasoning processes into executable structures or skills that can replace scratch-pad planning in later tasks.Examples include skill libraries, distilled skills and workflow graphs; hierarchical experiences can pair process knowledge with executable templates and check functions.Source: §3.3 · p. 9, §3.3.2 · p. 10

4 MEMORY CATEGORIZATION

The section traces memory from biological short- and long-term systems to a more granular agent taxonomy. It classifies agent memory by information type and by its applicability across tasks or sessions.

4 MEMORY CATEGORIZATION

  • Human memory stores and manages experiences, facts and skills for later access, while agent memory retains past interactions, including environmental records and interaction histories.The paper describes both as enabling information to remain available after the original stimulus or event.Source: §4 · p. 10

4.2 Memory Classification in Agents

  • Because the short- and long-term dichotomy cannot capture agent memory’s diversity and hierarchy, the paper proposes a more granular classification along two dimensions.The dimensions are nature-based, by encoded information, and scope-based, by applicability across tasks or sessions.Source: §4 · p. 10, Figure 2 · p. 11, §4.2 · p. 13

4.1.1 Short-term Memory.

  • Short-term memory temporarily maintains and manipulates limited information, generally for no more than 15–20 seconds and about 4–9 items.If information is not transferred to long-term storage before the short-term window closes, it is likely to be forgotten.Source: §4.1.1 · p. 11

4.1.2 Long-term Memory.

  • Long-term memory retains large amounts of information from minutes to decades, without a strict capacity limit, and supports retrieval and context for perception and decisions.It includes episodic memories of specific events and semantic memories of abstract knowledge; these systems interact and can transform into one another.Source: §4.1.2 · pp. 11–12

4.1.3 Distinction and Interaction.

  • Short- and long-term memory interact: short-term information may be consolidated, while long-term knowledge shapes how new information is encoded and maintained.Episodic memory relies strongly on the hippocampus and semantic memory primarily on the neocortex; repeated experiences can form semantic knowledge, which can guide episode reconstruction.Source: §4.1.3 · pp. 12–13

4.2.1 Nature-based Classification.

  • Nature-based classification distinguishes episodic memory for interaction trajectories and procedural experience from semantic memory for facts, concepts, rules and general knowledge.The distinction is framed as “how to do things” versus “what things are,” requiring experience- and knowledge-driven strategies to work together.Source: §4.2.1 · p. 13

4.2.2 Scope-based Classification.

  • Scope-based classification separates inside-trail memory, confined to one trajectory and typically reset afterward, from cross-trail memory that persists across executions.Cross-trail memory retains generalizable patterns and strategies for future episodes, whereas inside-trail memory holds temporary, task-specific context.Source: §4.2.2 · pp. 13–14

5 MEMORY STORAGE

The section compares biological and agent memory storage by location and format. It describes temporary and durable neural storage, then contrasts agents’ context windows and memory banks with text, graph, parametric and latent representations.

5.1.1 Short-term Memory Storage.

  • Human short-term memory relies on distributed sensory and frontoparietal regions and can be maintained through persistent neural activity or activity-silent synaptic states.The two formats may coexist: attended high-priority items remain active, while potentially useful unattended items can be stored silently and reactivated.Source: §5.1.1 · pp. 14–15

5.1.2 Long-term Memory Storage.

  • Long-term memory initially depends on hippocampal indexing of distributed neocortical traces, then gradually becomes sustained by neocortical networks through systems consolidation.The hippocampus also supports structured representations, including event-based units and cognitive maps; replay can integrate events and support cross-event structures.Source: §5.1.2 · pp. 15–17, Figure 3 · p. 16

5.2.1 Storage Location.

  • Agent context windows hold information within a reasoning trajectory, while external memory banks persistently store accumulated information across tasks or sessions.Context-window memory can be dynamically folded into summaries; memory banks retain reusable information beyond conversation termination or system restart.Source: §5.2.1 · p. 17

5.2.2 Storage Format.

  • Agent memory formats include natural-language text, graphs, model parameters and latent vectors, reflecting storage in symbolic, relational, weight-based or embedding forms.Text is interpretable and compatible with language models, while excessive compression can cause brevity bias in specialized domains [370].Source: §5.2.2 · p. 18
  • Graph memory represents entities and relationships as nodes and edges, supporting traversal for multi-step reasoning, relationship discovery and queries across experiences.The survey also describes graphs that connect multimodal memories across faces, voices and text through identity equivalence [189].Source: §5.2.2 · pp. 18–19
  • Parametric memory embeds experience and knowledge in model weights, enabling automatic activation during generation without retrieval latency.The survey describes imitation and reinforcement learning as routes to internalizing knowledge, strategies and operational skills [179] [12].Source: §5.2.2 · p. 19
  • Latent memory stores information as high-dimensional vectors, offering compact representations and trainable updates that can be integrated into model computation.Some methods iteratively retrieve and update latent vectors; an agent-memory approach triggers retrieval from the current action’s hidden state [362].Source: §5.2.2 · p. 19

6 MEMORY MANAGEMENT

Memory management is presented as a dynamic framework for regulating information across biological and artificial systems. It filters noise and preserves important patterns as raw perception becomes structured knowledge.

6 MEMORY MANAGEMENT

  • Memory management dynamically regulates the information lifecycle so systems can adapt to changing environments.The section frames this regulation across biological consolidation and artificial storage management.Source: §6 · p. 19
  • Effective management filters noise while preserving pivotal patterns, transforming raw perception into structured knowledge.The chapter analyzes this process from neuroscience foundations through information-flow management in autonomous agents.Source: §6 · pp. 19–20
  • The chapter compares memory management in biological brains with its systematic implementation in autonomous agents.It proceeds from neuroscientific foundations to information-flow management in agents.Source: §6 · p. 20

6.1 Memory Management in Cognitive Neuroscience

Long-term memory is a dynamic cycle in which formation, updating, and retrieval interact. Formation stabilizes and integrates experience, while updating and retrieval reshape existing traces and affect later learning.

6.1 Memory Management in Cognitive Neuroscience

  • Long-term memory management is a reciprocal cycle of formation, updating, and retrieval, rather than a one-way sequence.These processes interact with storage mechanisms and support flexible adaptation to the environment.Source: §6.1 · p. 20, Figure 4 · p. 20

6.1.1 Memory Formation.

  • Long-term memories form through encoding, hippocampal-neocortical consolidation, and integration into abstract, relational knowledge.Hippocampal replay helps stabilize new representations and link episodes over time.Source: §6.1.1 · p. 21

6.1.2 Memory Updating.

  • Prediction errors can trigger updating through differentiation, which preserves specificity, or integration, which organizes new information with prior memories.Integrative updating can preserve both factual corrections and memory of the original misinformation.Source: §6.1.2 · pp. 21–22

6.1.3 Memory Retrieval.

  • Partial contextual cues can trigger hippocampal pattern completion, while retrieval flexibly replays episodes and reinstates details in the neocortex.Replay direction can depend on the task, such as recalling items before or after a cue.Source: §6.1.3 · p. 22
  • Retrieval can make existing traces labile for strengthening, weakening, or updating, and can prime encoding of subsequent information.If subsequent input is inaccurate, retrieval-related priming may also expedite internalization of misleading information.Source: §6.1.3 · pp. 22–23

6.2 Memory Management in Agents

Agent memory management is a closed-loop process that extracts, updates, retrieves, and applies experience. The survey groups its methods into distinct paradigms at each stage, from memory construction to use in decisions.

6.2 Memory Management in Agents

  • Agent memory management forms a closed loop of extraction, updating, retrieval, and utilization, supporting persistent experience regulation and long-range reasoning.Extraction structures interaction streams; updating and retrieval maintain validity and locate relevant knowledge, while utilization applies it to decisions.Source: §6.2 · p. 23, Figure 5 · p. 23

6.2.1 Memory Extraction.

  • Memory extraction methods are flat, hierarchical, or generative, differing in how they structure or reconstruct useful content from interaction streams.Flat methods record or lightly process information; hierarchical methods organize it across levels; generative methods reconstruct context during reasoning.Source: §6.2.1 · pp. 23–24

6.2.2 Memory Updating.

  • Agent updating operates inside trials to refresh working context and across trials to maintain long-term archives through selective retention, forgetting, and integration.Cross-trial approaches also include learned retention policies and autonomous goal-driven knowledge reconstruction.Source: §6.2.2 · p. 25

6.2.3 Memory Retrieval.

  • Current agent retrieval is mainly selective activation: methods use semantic similarity or combine factors such as recency, importance, structure, and expected reward.Similarity retrieval can miss procedural logic, while multi-factor approaches prioritize memories beyond semantic matching alone.Source: §6.2.3 · pp. 25–26

6.2.4 Memory Application.

  • Memory application uses contextual augmentation or parameter internalization to bring accumulated experience into reasoning and model behavior.Contextual methods dynamically assemble or integrate memory with reasoning, while internalization distills explicit memory into model parameters.Source: §6.2.4 · pp. 26–27

7 BENCHMARKS FOR AGENT MEMORY

The survey organizes agent-memory benchmarks around semantic state maintenance and episodic performance in applied domains. It describes evaluation through fidelity, dynamics, and generalization, with episodic tests emphasizing complex task execution and environmental conditions.

7 BENCHMARKS FOR AGENT MEMORY

  • Agent-memory benchmarks are grouped as semantic-oriented or episodic-oriented, and assessed along fidelity, dynamics, and generalization.The dimensions cover accurate recall, memory updating, and translating stored information into behavior suited to new environments.Source: §7 · p. 27, Table 1 · p. 28, Table 2 · p. 29

7.1 Benchmarks for Semantic-oriented

  • Semantic-oriented benchmarks test internal-state construction, maintenance, retrieval fidelity, updating, and self-evolution.The benchmarks range from retaining details across long contexts to abstracting rules from errors and transferring them to novel scenarios.Source: §7 · p. 27, §7.1 · pp. 27–28

7.2 Benchmarks for Episodic-oriented

  • Episodic-oriented benchmarks assess memory use in web search, tool use, and environmental interaction, including state tracking under noisy real-world conditions.Examples test navigation, tool-schema retrieval, multi-step environmental reasoning, and adapting to user history; the survey frames this category as shifting evaluation toward problem solving.Source: §7.2 · pp. 28–30

8 SECURITY OF AGENT MEMORY

Agent memory supports continuity in long-term planning and tool use, but also creates channels for privacy leaks and malicious memory injection. The survey organizes attacks and defenses across memory’s lifecycle.

8.1 Attack of Agent Memory

  • Extraction-based attacks steal user data, while poisoning-based attacks inject toxic information to compromise agent decisions.The survey identifies privacy leakage and malicious memory injection as distinct security threats.Source: §8.1 · p. 30

8.1.1 Extraction-based Attack.

  • Extraction attacks span private-database reconstruction [115], structured RAG prompting [355], and black-box elicitation of agents’ stored interaction histories [274].The cited work traces attacks from retrieval models to complex agent workflows.Source: §8.1.1 · p. 30

8.1.2 Poisoning-based Attack.

  • Poisoning injects optimized malicious data into external memory without changing model parameters, using stealthy triggers to activate backdoors while routine behavior remains normal.Retrieval preferences and reliance on context can be exploited to hijack decision-making.Source: §8.1.2 · pp. 30–31
  • Cognitive poisoning adds noise, conflicting information, or social bias to memory, potentially degrading judgment or distorting values [66].Examples include ineffective recommendations [333] and persistent discriminatory outputs [11].Source: §8.1.2 · p. 31

8.2 Defense of Agent Memory

  • The survey groups defenses into retrieval-based purification, response-based blocking, and privacy safeguards across the memory lifecycle.Together, these mechanisms form a closed-loop defense system.Source: §8.2 · p. 31

8.2.1 Retrieval-based Defense.

  • Retrieval-based defenses detect anomalies and filter poisoned records before retrieved knowledge enters explicit memory [240].Proposals include consensus validation of multiple memories [295] and dual-agent detection and repair of poisoned code-generation reasoning [134].Source: §8.2.1 · p. 31

8.2.2 Response-based Defense.

  • Response-based defenses block malicious logic after it enters memory, using collaborative review [356] or trajectory rehearsal and scoring [394].These mechanisms operate during response generation.Source: §8.2.2 · p. 31

8.2 Defense of Agent Memory

  • Privacy-based defenses isolate sensitive data and use only anonymized or task-necessary memory for reasoning and collaboration [101] [234].Examples anonymize shared workspaces [329] or remove unnecessary sensitive context while retaining task intent [356].Source: §8.2.3 · p. 32

9 FUTURE DISCUSSION

The survey highlights multimodal memory and transferable agent skills as future research directions. It describes current approaches, their constraints, and open challenges for both directions.

9.1 Multimodal Memory

  • Multimodal memory work converts raw inputs into symbolic summaries, compresses representations, or pairs symbolic memories with aligned multimodal evidence.Examples include timestamps, captions, and object categories, alongside token merging and Q-Former compression.Source: §9.1 · p. 32
  • Multimodal memory systems report improved performance across environments, supporting interactive planning, long-video understanding, audio–visual comprehension, personalization, and more efficient GUI interaction.The survey cites systems spanning games, video, audio–visual streams, multimodal dialogue, and GUI trajectories.Source: §9.1 · p. 32
  • Seamlessly adapting memory operations across modalities while preserving semantic consistency and temporal alignment remains unsolved.Compression or abstraction can degrade semantics; long-term temporal dependencies and computational efficiency also constrain scalability.Source: §9.1 · pp. 32–33

9.2 Agent Skills

  • A test-time memory-trigger approach adapts through fast parameters, but its complexity and agent-confined learned parameters limit transfer of specialized knowledge.The survey argues for memory strategies with greater composability, scalability, and portability.Source: §9.2 · p. 33
  • Agent skills package instructions, executable scripts, and resources into modular units that agents can discover and load at runtime for domain-specific tasks.The modules are intended to make expertise composable, reusable, and shareable across agents.Source: §9.2 · p. 33
  • Open skill-sharing challenges include unified multimodal storage and cross-modal retrieval, plus transferring and adapting skills across heterogeneous agent architectures.The survey also notes early work on portable memory interfaces and shared-memory platforms.Source: §9.2 · p. 33

10 CONCLUSION

The survey unifies memory research across cognitive neuroscience and autonomous agents, covering core mechanisms and emerging directions. It identifies a disciplinary gap and calls for cross-disciplinary collaboration to develop more robust agent memory systems.

10 CONCLUSION

  • The survey investigates memory in cognitive neuroscience and autonomous agents, covering definitions, functions, taxonomies, management, security, and future directions.Its scope spans the paper’s main topics rather than a single memory mechanism.Source: §10 · p. 33
  • Existing work has not deeply drawn on brain-science memory mechanisms, which the survey attributes to a gap between disciplines.The conclusion presents this gap as a constraint on integrating human-like memory mechanisms into agents.Source: §10 · p. 34
  • The authors envision cross-disciplinary collaboration as a step toward more robust agent memory systems.Source: §10 · p. 34
Loading 2512.23343v1…