PaperlayerView exact revision on arXiv

Source-linked AI summary

The AI Hippocampus: How Far are We From Human Memory?

Zixia Jia, Jiaqi Li, Yipeng Kang, Yuxuan Wang, Tong Wu, Quansen Wang, Xiaobo Wang, Shuyi Zhang, Junzhe Shen, Qing Li, Siyuan Qi, Yitao Liang, Di He, Zilong Zheng, Song-Chun Zhu

arXiv:2601.09113v1cs.AI

TL;DR

The survey addresses the lack of a unified account of memory across LLMs, agents, and multimodal models. It synthesizes implicit, explicit, and agentic memory paradigms, reviews multimodal mechanisms and empirical evaluations, and identifies future research directions and scope limitations.

  • Problem

    Existing surveys do not unify memory studies across pure LLMs, LLM-based agents, and multimodal models.

  • Method

    The paper presents a narrative review of implicit, explicit, and agent memory in LLMs, alongside memory mechanisms designed for multimodal LLMs.

  • Results

    The survey maps the LLM-memory landscape, traces its development, and covers milestones, empirical evaluations, and forward-looking research directions.

  • Takeaways & Limitations

    Future work should improve implicit-memory modeling and long-context processing while accounting for computational-efficiency trade-offs.

  • Takeaways & Limitations

    The survey lacks a unified evaluation framework for all memory types and does not propose a single platform integrating them.

Abstract

from arXiv · show

Memory plays a foundational role in augmenting the reasoning, adaptability, and contextual fidelity of modern Large Language Models and Multi-Modal LLMs. As these models transition from static predictors to interactive systems capable of continual learning and personalized inference, the incorporation of memory mechanisms has emerged as a central theme in their architectural and functional evolution. This survey presents a comprehensive and structured synthesis of memory in LLMs and MLLMs, organizing the literature into a cohesive taxonomy comprising implicit, explicit, and agentic memory paradigms. Specifically, the survey delineates three primary memory frameworks. Implicit memory refers to the knowledge embedded within the internal parameters of pre-trained transformers, encompassing their capacity for memorization, associative retrieval, and contextual reasoning. Recent work has explored methods to interpret, manipulate, and reconfigure this latent memory. Explicit memory involves external storage and retrieval components designed to augment model outputs with dynamic, queryable knowledge representations, such as textual corpora, dense vectors, and graph-based structures, thereby enabling scalable and updatable interaction with information sources. Agentic memory introduces persistent, temporally extended memory structures within autonomous agents, facilitating long-term planning, self-consistency, and collaborative behavior in multi-agent systems, with relevance to embodied and interactive AI. Extending beyond text, the survey examines the integration of memory within multi-modal settings, where coherence across vision, language, audio, and action modalities is essential. Key architectural advances, benchmark tasks, and open challenges are discussed, including issues related to memory capacity, alignment, factual consistency, and cross-system interoperability.

1 Introduction

Memory is presented as a central capability for transforming LLMs from static predictors into adaptive, context-aware systems. The survey organizes this landscape across implicit, explicit, and agentic memory, while extending the framework to multimodal models.

  • Motivation: Memory modules let LLMs accumulate knowledge, process historical experiences, and retrieve information for decision-making and actions.They support continuous learning, adaptation, long-term planning, and more personalized responses.
  • Memory Framework: The survey distinguishes implicit memory in model parameters, explicit memory in external storage and retrieval, and agentic memory for persistent environmental interactions.These paradigms are framed respectively through analogies to the neocortex, hippocampus, and prefrontal cortex.
  • Implicit Memory: Implicit memory corresponds to the stable, generalized knowledge embedded in transformer weights during pre-training.The survey associates these parameters with distributed semantic knowledge, language patterns, and facts.
  • Explicit Memory: Explicit memory systems such as Retrieval-Augmented Generation provide queryable access to specific, real-time, or episodic information without retraining the full model.They function as an index for grounding responses in up-to-date facts.
  • Agentic Memory: Agentic memory maintains persistent state, manages working memory, and strategically retrieves implicit and explicit information to plan and execute complex tasks.Its executive function can integrate specialized modules for spatial, temporal, and embodied intelligence.
  • Survey Scope: Existing surveys do not provide a unified view spanning pure LLMs, LLM-based agents, and multimodal models, which this survey aims to provide.The review connects memory mechanisms across language, agentic, and multimodal settings.

2 Implicit Memory: Unveiling Knowledge Inside Transformers

Implicit memory is the information embedded in Transformer parameters, supporting memorization, associative retrieval, and contextually relevant responses. Research examines how this knowledge is stored, retrieved, modified, and evaluated, while highlighting limits in capacity, efficiency, generalization, and unlearning safety.

  • Definition: Implicit memory is intrinsic information embedded in model parameters, including facts, commonsense, self-knowledge, and associative memory.
  • Knowledge Memorization: Transformer research studies how parameters and components such as FFNs and self-attention contribute to implicit knowledge memorization and retrieval.
  • Knowledge Memorization: Fact memorization capacity scales with model size and training, but memorizing all public facts remains nearly impossible in general pre-training.
  • Associative Memory: Associative-memory studies link error rates to model capacity and training inputs, with finite capacity causing interference when new associations remap existing memory locations.
  • Implicit Memory Modification: Implicit memory modification comprises incremental training, memory editing, and memory unlearning, which respectively add, alter, or remove parameterized knowledge.
  • Limitations and Open Questions: Current work is constrained by task-specific evidence, costly component probing, and the risk that unlearning disrupts related knowledge and downstream performance.

3 Explicit Memory: When (M)LLMs Meet Retrieval

Explicit memory stores factual knowledge and historical trajectories externally so retrievers can capture context-aware information and generators can incorporate it across sessions. The survey organizes explicit memory by representation granularity and structure, including documents, chunks, sentences, graphs, triples, and vectors.

  • Explicit memory stores factual knowledge and historical trajectories in external structured or unstructured representations.It supports dynamic retrieval and incorporation of external context rather than relying only on model parameters.
  • The section examines how explicit memory is represented, updated during training, and used to externalize implicit knowledge for retrieval.
  • Free text: Documents preserve rich context but can introduce noise, increase inference costs, and exceed context windows when overly long.Long inputs may obscure specific facts or be truncated, risking omission of critical data.
  • Free text: Chunks reduce redundancy and strengthen query-memory connections, while top-k selection addresses the infeasibility of using every relevant chunk.RetrievalML selects top-k chunks by relevance, whereas PaperQA uses LLM-generated relevance scores.
  • Free text: Sentence-level units capture specific facts and details but can overlook inter-sentence connections, making them suitable for detail-focused tasks.The passage identifies sentiment analysis and knowledge editing as examples.
  • Structured representations: Graph-based representations support logical deduction and multi-hop reasoning, while triples provide fine-grained facts but may fragment continuous semantic information.Sub-graphs retrieve query-related portions of a larger graph and can expose reasoning paths.
  • Vector representations: Vector representations encode fragments continuously and retrieve the top K by query-embedding similarity, offering semantic understanding and scalable indexing.Vectors can use efficient indexing, quantization, and batch processing compared with keyword matching and heavier preprocessing.

3.2 Training with Explicit Memory

Training with explicit memory teaches models to retrieve, adapt, and integrate external representations rather than relying solely on similarity or structural matching. The survey covers pre-training and fine-tuning approaches, including retrieval integration, task specialization, self-reflection, and efficiency-oriented designs, while identifying computational, interpretability, and autonomous-training challenges.

  • Training with explicit memory teaches models to retrieve, organize, and integrate external representations instead of relying solely on similarity or structural matching.This can reduce redundancy in memory use.
  • Training can improve retrieval relevance and generation accuracy by aligning modules for context-sensitive use and integrating past experiences with model reasoning.The passage specifically highlights long-context, consistency, and multi-turn interaction scenarios.
  • The survey distinguishes pre-training and fine-tuning as two phases for improving explicit-memory systems.
  • Challenges: Explicit-memory training faces high real-time encoding costs, limited retrieval interpretability, and poor retrieval or generation in autonomous training.
  • Pre-training: Unsupervised retrieval pre-training jointly models retrieval and prediction, while RETRO integrates retrieved chunks through cross-attention to expand memory capacity.REALM demonstrated improvements on open-domain question answering; RETRO freezes a pre-trained BERT retriever during pre-training.
  • Pre-training: Other pre-training methods target efficiency and knowledge-intensive tasks through selective key-value storage, dual-encoder retrieval, and joint sequence-to-sequence training.Memory3 is described as reducing read and write costs while improving efficiency and performance across tasks.
  • Fine-tuning: Fine-tuning specializes explicit memory for task- or domain-specific retrieval, while RAG jointly fine-tunes retriever and generator to condition outputs on external knowledge.RAG is reported to improve factuality, diversity, and specificity in generated responses.
  • Fine-tuning: Self-RAG adds reflection tokens so the generator learns when and how to retrieve, while other methods target prompt retrieval, subgraph retrieval, and efficient passage processing.FiD processes passages independently in the encoder and jointly in the decoder.

3.3 Training with externalized parameteric knowledge

Externalized parametric knowledge stores model knowledge or intermediate outputs outside the model for retrieval, supporting long-context processing and knowledge injection. The section also identifies unresolved challenges in retrieval quality, memory consistency, and scalability.

  • Core approach: Externalized parametric knowledge extracts and stores internal knowledge or intermediate outputs in structured, accessible formats.This supports retrieval when the model’s internal memory is insufficient.
  • Long-context motivation: Transformer attention has limited context length, making distant information difficult to retain for tasks requiring long-range dependencies.MemTRM addresses this by memorizing representations of previous inputs.
  • Long-context methods: Unlimiformer offloads cross-attention computation to a kNN index, enabling layer-specific top-k retrieval without fine-tuning.Its non-parametric design lets each attention head focus on selected keys.
  • Long-context methods: LongMem addresses distribution shift by decoupling the memory encoder from the adaptive residual network.The original language model is frozen as the memory encoder.
  • Knowledge injection: Retrieval-based knowledge injection can fold contextual information into model parameters for downstream reasoning.TRIME aligns token representations with embeddings and contextualized memory representations, while TOME and OPQL construct retrievable text representations.
  • Open challenges: Open questions concern whether retrieval can resolve long-context limits, how to retrieve autonomously, and how to prevent hallucination and memory contamination.The survey also highlights consistency between external and implicit memory and the computational burden of large-scale retrieval training.

4 Agentic Memory: Consolidating Memories into Humanic Agents

Agentic memory applies human memory concepts to LLM agents, distinguishing transient context-based memory from externally stored long-term memory. The survey identifies a gap in comprehensive cognitive and engineering-level coverage and organizes its review around agent memory operations and evaluation.

  • Motivation: LLM agents use memory systems to accumulate experiences, adapt continuously, and support consistent, rational, and effective behavior.These systems are presented as mirroring aspects of human memory.
  • Human-memory analogy: Sensory memory is treated as the AI data-ingestion pipeline and is not discussed in detail.The analogy includes systems such as DataLoader.
  • Memory types: Short-term memory corresponds to information maintained in the context window during in-context learning and is constrained by Transformer context length.Long-term memory is represented by externally stored information accessed through retrieval.
  • Research gap: The survey notes that a comprehensive categorization grounded in human cognitive processes and a review of engineering-level agent memory systems were missing from prior work.This gap motivates the section’s broader treatment of agentic memory.
  • Survey scope: The review covers short- and long-term operation, multi-agent memory sharing, ingestion through application, and evaluation methodologies.These topics define the section’s scope.

4.1 Single-agent Memory

Single-agent memory combines external recall, post-response memory updates, short-term reasoning traces, and long-term stores for facts, experiences, feedback, and personalization. The survey organizes these mechanisms across memory formation, retrieval, and evolution.

  • External memory: External-memory agents commonly recall relevant thoughts before responding and incorporate historical and new thoughts afterward.This two-stage process is intended to improve consistency and efficiency.
  • External memory: The section synthesizes inference-time RAG methods that use external memory to improve downstream agent tasks.The focus is on refined and optimized retrieval for agents.
  • Short-term memory: Short-term memory retains recent inputs and reasoning traces in context, improving continuity but discarding history beyond the context window.CoT stores intermediate reasoning steps as short-term memory, while ReAct interleaves reasoning with actions.
  • Long-term memory: Long-term memory is organized into fact knowledge, historical trajectories and experience, user feedback, and dialogues with personalized configuration.These categories are presented as the section’s four primary long-term-memory types.
  • Long-term memory: Historical experience helps agents identify patterns, make informed predictions, and adapt strategies based on past data.ReasoningBank preserves transferable strategies while abstracting away low-level execution details.
  • Long-term memory: Long-term memory can preserve user corrections, clarifications, dialogue information, and personalized knowledge for future reference.Personalized knowledge bases support retrieval through programmatic interaction with stored knowledge.

4.2 Multi-agent Memory

Multi-agent systems use shared memory to reuse historical information and coordinate actions, but must balance information sharing with agent autonomy and overload. Implementations include shared vector repositories and communication frameworks for heterogeneous agents.

  • Shared memory: Shared memory lets collaborating agents query past interactions and knowledge to inform current actions.This can improve reasoning and coordination across agents.
  • Design trade-offs: Shared-memory designs trade off real-time experience reuse against autonomy, flexibility, robustness, and information overload.The central challenge is deciding how much information agents should exchange.
  • Implementation: Vector-based shared repositories commonly retrieve relevant memories using similarity metrics.MS stores prompt-answer pairs in a shared pool and uses an autonomous retriever.
  • Implementation: IoA supports heterogeneous-agent integration through discovery, dynamic team formation, and structured communication.Its framework is modeled on Internet-style instant messaging.

4.3 System Architecture

The surveyed memory-augmented agent architectures organize external memory through ingestion, storage and retrieval, and user-facing application interfaces. They use vector, graph, hybrid, and persistent storage strategies, with retrieval tailored to application needs and constrained by scalability and context limits.

  • Architecture: External-memory systems typically follow a Data Ingestion → Storage and Retrieval → User Interfaces and Application Invocation pipeline.The survey compares these modules across mainstream agent-system architectures.
  • Data ingestion: Data connectors ingest local files, web content, databases, APIs, streaming data, conversations, tools, and domain-specific documents.Conversational systems may also store user messages, model responses, tool outputs, and inter-agent messages.
  • Storage: Vector storage supports large-scale semantic retrieval by matching query embeddings against vectors produced from segmented text.Frameworks may use local or external vector databases for persistence and distributed capabilities.
  • Storage: Graph databases represent entities and relationships for structured reasoning, relationship tracing, version control, and dynamically updated retrieval.Graph-based systems can invalidate outdated relationships as their stored information changes.
  • Storage: Hybrid and persistent storage balance speed, capacity, and restart durability by combining immediate context with external archives.Without persistence, in-memory vector indices and memories are lost after restart.
  • Retrieval and interfaces: Retrieval methods differ by application, while truncation and filtering keep retrieved memories within model context windows; simpler databases struggle with complex scaling.Developer APIs generally expose memory through background services, logs, or dashboards, while some platforms provide web interfaces.

4.4 Evaluation on Agent Memory

Agent-memory evaluation should measure both final task outcomes and intrinsic memory behavior across static, interactive, and communicative settings. The surveyed framework includes task success and efficiency alongside capabilities such as recall, multi-session reasoning, temporal reasoning, and abstention.

  • Evaluation framework: Agent-memory evaluation spans final task outcomes and intrinsic memory properties across diverse tasks and granularities.The survey presents a framework intended to organize this evaluation landscape.
  • Evaluation tasks: Long-text and question-answering benchmarks test retention, retrieval, and reasoning over lengthy contexts and long-range dependencies.Examples include NarrativeQA, QuALITY, and Loogle.
  • Evaluation tasks: Interactive and open-world benchmarks assess memory during dynamic, multi-step decisions requiring planning, adaptation, learning, and interaction with environments.AgentBench evaluates agents across environments including operating systems and games.
  • Metrics: Metrics operate at general-purpose, low-level intrinsic, and specialized communicative-learning levels.This separates overall task performance from the memory module’s fundamental capabilities and communication behavior.
  • Metrics: Task success and accuracy measure completed tasks or final-answer correctness, while latency and cost capture efficiency for practical deployment.The survey identifies task success and accuracy as the primary overall capability indicator.
  • Metrics: Intrinsic capabilities include recall and precise fact extraction, multi-session synthesis, knowledge utilization, temporal reasoning, and abstention on unknown information.These measures test whether agents use stored information accurately across histories and time.

Comparative Evaluation of Memory Frameworks on General and Low-Level Metrics

The comparative evaluation examines memory frameworks across correctness, processing time, learning efficiency, generalization, controllability, and robustness. Results favor simple or balanced systems in several settings, while multi-session reasoning remains difficult and memory perturbations strongly affect performance.

  • General evaluation: The comparison evaluates no-memory and multiple external-memory frameworks using Llama3-8B-IT and GPT-4o-mini, measuring correctness and total processing time.Total time includes data ingestion, retrieval, and reasoning, with correctness evaluated using GPT-4o-mini.
  • General evaluation: Ingestion and retrieval constitute most processing time for every framework except the no-memory baseline.This pattern holds for the reported framework evaluations.
  • Framework comparison: ChromaDB performs surprisingly well, while Haystack and LlamaIndex show a strong balance of performance and efficiency.Several more complex systems do not produce the improvements suggested by their documentation; LangChain is reported as least effective.
  • Framework comparison: Multi-session reasoning is the most challenging task across frameworks because each question averages 47 sessions with around 10 conversational turns per session.ChromaDB benefits on single-session tasks, whereas more advanced systems show a slight advantage on demanding multi-session tasks.
  • Low-level evaluation: RAG and RAM performance basically improves consistently as the proportion of necessary memories increases from 10% to 100%.Figure 12 compares memory settings across varying quantities of context needed to answer each question.
  • Low-level evaluation: Memory perturbations significantly influence generalization, controllability, and robustness, while generalization of past memories is not observed, potentially because of insufficient data volume.The evaluation also includes functionality and perturbation tests, with functionality covering learning efficiency and generalization and perturbation covering controllability and robustness.

4.5 Limitations, Open Questions, Discussion

The survey positions itself as a broad account of memory across LLMs, agents, and multimodal models, while identifying dynamic adaptation and communication as important open directions. It emphasizes system-level integration across modalities and deployment-oriented memory design.

  • Scope and positioning: The survey distinguishes its scope from agent-focused and mechanistic reviews by covering LLMs, agents, and multimodal models in one system-level perspective.It frames memory as foundational to modern LLMs and MLLMs, including integration across text, vision, and audio.
  • Open directions: Dynamic memory adaptation during reasoning and communicative learning is identified as a potentially crucial direction for improving agent utility.RAM is cited as demonstrating recursive retrieval and experience reflection for continuous memory updates based on user feedback.

5 Memory-augmented Multi-Modal Large Language models

Memory mechanisms help multimodal systems preserve and use long-context information across audio, video, and embodied tasks. The survey organizes these applications around contextual modeling, memory-enhanced representations and agents, downstream video tasks, and unresolved scaling challenges.

  • Multimodal Context Modeling: Multimodal memory combines audio, video, and other modalities to model context for diverse downstream tasks.The survey frames this as a response to the need to preserve extensive multimodal context for complex datasets and embodied agents.
  • Audio Context Modeling: Audio history is difficult to model efficiently because continuous, high-frequency sequences require substantial computational resources.Conformer-NTM places an external memory network between the encoder and decoder, preventing instrument leakage in automatic music transcription.
  • Video Context Modeling: Video memory mechanisms address the complexity of temporal context by consolidating information across segments and retrieving semantically relevant frame features.MC-ViT uses memory consolidation and cross-attention for long-video encoding, while other systems target video question answering and long-video modeling.
  • Memory-enhanced Video Agents: Memory supports video agents by storing textual representations such as captions, object names, and timestamps for language-model-based video processing.Related systems combine captioning, tracking, and visual question answering to inject video understanding capabilities into language models.
  • Downstream Tasks: Video memory is applied across object-level, action-level, and general video understanding tasks, including tracking, segmentation, classification, localization, and anticipation.Action-level systems selectively retain significant information or organize long- and short-term memory to preserve relevant temporal context.
  • Limitations and Open Questions: Multimodal systems face scaling challenges from visual-token abundance, multi-turn reasoning, time-series dependencies, and the need to memorize embodied interactions.The central frontier is balancing computational efficiency with model effectiveness while retaining long-term multimodal context.

6 Conclusion

The survey reviews memory in LLMs, agents, and multimodal LLMs through three categories: implicit memory in model parameters, explicit external storage and retrieval, and agent memory for persistent environmental interactions.

  • Conclusion: The survey presents a narrative review of implicit, explicit, and agent memory in LLMs, alongside memory mechanisms designed for multimodal LLMs.It traces memory mechanisms from foundational concepts through recent advances and discusses empirical evaluations and future directions.

7 Future Work and Limitations

The survey identifies future work in implicit-memory modeling, long-context processing, dynamic adaptation, and scalable multimodal memory. It also states that the survey lacks a unified evaluation framework and an integrated platform spanning these memory types.

  • Future Directions: Future work should improve understanding and modeling of implicit memory in Transformer architectures.The survey identifies this as a critical research need.
  • Future Directions: Long-context processing requires extended context windows or retrieval-augmented generation, with trade-offs in computational efficiency and scalability.The survey presents both approaches as important but constrained by different resource considerations.
  • Future Directions: Dynamic memory adaptation inspired by recursive retrieval and experience reflection is identified as promising for reasoning and communication in agent-based systems.The survey connects this direction to human learning strategies.
  • Limitations: The survey does not provide a unified evaluation framework for all memory types or propose a single platform integrating them.This limitation reflects the diverse aspects emphasized by different memory mechanisms.
Loading 2601.09113v1…