Source-linked AI summary
MemVerse: Multimodal Memory for Lifelong Learning Agents
Junming Liu, Yifei Sun, Weihua Cheng, Haodong Lei, Yirong Chen, Licheng Wen, Xuemeng Yang, Daocheng Fu, Pinlong Cai, Nianchen Deng, Yi Yu, Shuyue Hu, Botian Shi, Ding Wang
TL;DR
AI agents often operate in a stateless way, limiting memory for continuous interaction and long-horizon reasoning. MemVerse combines parametric recall with structured multimodal retrieval and periodic distillation. Across diverse multimodal tasks, it improves reasoning accuracy, stability, and long-horizon coherence.
Problem
Agents often treat tasks as isolated episodes, despite memory’s importance for integrating experience, adapting to dynamic environments, and long-horizon reasoning.
Source: §1 · p. 1Method
MemVerse combines fast parametric recall with hierarchical graph-based memory and periodically distills essential long-term knowledge into a lightweight parametric model.
Source: §1 · p. 2Results
Across diverse multimodal tasks, MemVerse yields significant improvements in reasoning accuracy, stability, and long-horizon coherence.
Source: §5 · p. 8Takeaways & Limitations
The paper concludes that scalable memory, rather than ever-larger models, is the missing ingredient for continuous-learning agents.
Source: §5 · p. 8The graph-based memory organization has so far been evaluated primarily in small- to medium-scale memory settings, where longer interactions may raise retrieval and insertion overhead and introduce outdated-memory noise.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Despite rapid progress in large-scale language and vision models, AI agents still suffer from a fundamental limitation: they cannot remember. Without reliable memory, agents catastrophically forget past experiences, struggle with long-horizon reasoning, and fail to operate coherently in multimodal or interactive environments. We introduce MemVerse, a model-agnostic, plug-and-play memory framework that bridges fast parametric recall with hierarchical retrieval-based memory, enabling scalable and adaptive multimodal intelligence. MemVerse maintains short-term memory for recent context while transforming raw multimodal experiences into structured long-term memories organized as hierarchical knowledge graphs. This design supports continual consolidation, adaptive forgetting, and bounded memory growth. To handle real-time demands, MemVerse introduces a periodic distillation mechanism that compresses essential knowledge from long-term memory into the parametric model, allowing fast, differentiable recall while preserving interpretability. Extensive experiments demonstrate that MemVerse significantly improves multimodal reasoning and continual learning efficiency, empowering agents to remember, adapt, and reason coherently across extended interactions.
1 Introduction
MemVerse addresses agents’ limited memory by coordinating parametric recall with retrieval-based memory. Its design turns multimodal experience into structured memories and periodically distills long-term knowledge for fast recall.
1 Introduction
- Agents often treat tasks as isolated episodes, a bottleneck for continuous interaction, adaptation, and long-horizon reasoning.The introduction describes current agents as largely stateless, unlike human memory’s continuous accumulation and abstraction of multimodal experience.Source: §1 · p. 1
- Parametric memory can suffer fixed capacity, costly retraining, catastrophic forgetting, and limited interpretability, while raw-log retrieval can accumulate redundancy, noise, and computational cost.The introduction argues that these limitations arise from the paradigms’ isolated use, since they address different aspects of memory.Source: §1 · p. 1
- MemVerse coordinates fast parametric recall and slow graph-based retrieval as complementary pathways, with adaptive routing between them.The introduction frames the limitation as using either pathway in isolation: parametric memory supports compact generalization, while retrieval preserves explicit, revisitable experiences.Source: §1 · pp. 1–2
- The framework transforms raw multimodal experiences into hierarchical knowledge graphs, enabling continual abstraction, adaptive forgetting, and bounded memory growth.Its graph structure organizes core, episodic, and semantic memories and is continuously updated from agent experience.Source: §1 · p. 2
- Periodic distillation compresses essential long-term memory into a lightweight parametric model for fast, differentiable recall while preserving transparency and controllability.Source: §1 · p. 2
2 Related Work
Prior work spans parametric and external-retrieval memory for LLM agents, alongside multimodal retrieval and graph-based methods. The section identifies interaction inefficiency and inflexible long-term retention as remaining concerns.
2.1 Memory for LLM Agents
- Parametric approaches encode knowledge in model parameters, while external-storage systems use retrieval or hierarchical mechanisms; the section notes interaction inefficiency and limited flexible retention as concerns.Source: §2.1 · pp. 2–3
2.2 Multimodal Knowledge Retrieval
- Multimodal retrieval work includes varied retrieval strategies, while knowledge-graph methods typically use graphs as static external knowledge bases indexed for retrieval.Source: §2.2 · p. 3
3 Methodology
MemVerse combines short- and long-term retrieval memory with a periodically distilled parametric memory, while an orchestrator manages memory operations.
3 Methodology
- The orchestrator manages memory addition, update, retrieval, consolidation, and deletion across short-term, long-term, and parametric memory.Parametric memory supplies fast, differentiable recall and generalization through compact representations distilled from long-term memory.Source: §3 · p. 3
3.1 Hierarchical Retrival-based Memory
- Pretrained multimodal models convert inputs such as images, videos, and audio into textual representations for memory processing.Modality-specific encoders and a projection map representations into a unified text-token space, while metadata links text chunks back to preserved raw inputs.Source: §3.1 · p. 3
- Short-term memory caches recent dialogue queries in a sliding window, with consolidation deferred until periodically or when enough new knowledge accumulates.This avoids unnecessary frequent updates to external memory.Source: §3.1 · pp. 3–4
- Long-term memory pairs knowledge graphs for core, episodic, and semantic memories with the original dialogue text chunks.Raw dialogue is compressed into salient descriptions and routed by user dependency, temporal scope, and abstraction level.Source: §3.1 · p. 4
- An LLM extracts entities and typed relations from text chunks, while graph nodes and relations retain references to supporting chunks for retrieval.Activating a graph entity or relation can retrieve its supporting dialogue and associated multimodal data.Source: §3.1 · p. 4
3.2 Lightweight Parametric Memory Model
- The lightweight parametric model is fine-tuned on question-and-answer examples derived from long-term-memory retrieval to emulate retrieval behavior.The objective is token-level cross-entropy between generated and target answers, embedding retrieved knowledge in model parameters.Source: §3.2 · pp. 4–5
- New query–evidence pairs from explicit memory drive lightweight, localized parameter updates as the knowledge graph expands, without full retraining.The parametric memory remains a fast-access complement to, rather than a sole store replacing, retrieval-based memory.Source: §3.2 · p. 5
4 Experiment
MemVerse is evaluated on long-term memory, multimodal reasoning, and video-text retrieval benchmarks, with ablations and efficiency analyses probing its components. Results report strong benchmark performance, while identifying reader reasoning and open-domain retrieval as constraints.
4.1 Setups
- The evaluation covers LoCoMo, LongMemEval, ScienceQA, and MSR-VTT, comparing against language models and memory, reasoning, and retrieval baselines.The reported LongMemEval and LoCoMo overall scores were recalculated under the official LoCoMo weighted evaluation protocol for methods marked with an asterisk.Source: §4.1 · pp. 5–6
4.2 Main Results
- On LoCoMo, MemVerse achieves state-of-the-art Overall F1 across four backbones, reaching 60.0 with GPT-3.5-turbo-16k and outperforming RAG by a large margin.The reported gains are more modest with Qwen2.5-7B-Instruct.Source: §4.2 · p. 6, Table 1 · p. 6
- On LongMemEval, Rec@5 is 89.8% and NDCG@5 is 91.6%, alongside 68.4% downstream accuracy; GPT-5.1 raises accuracy to 77.0%.The authors attribute the initial divergence between retrieval and answer accuracy to generator over-reasoning, rather than memory-pipeline quality.Source: §4.2 · p. 7, Table 2 · p. 7
- ScienceQA accuracy reaches 85.48% with GPT-4o-mini; on MSR-VTT, T2V R@1 is 90.4% and V2T R@1 is 89.2%.The MSR-VTT scores surpass the previous best-performing baseline ExCae by more than 20%.Source: §4.2 · p. 7, Table 3 · p. 7, Table 4 · p. 7
4.3 Ablation Study
- On LoCoMo, +Long-Term raises Overall F1 from 11.1 to 39.9 with GPT-4o-mini and from 23.3 to 32.1 with Qwen2.5-7B-Instruct; +Orchestrator reaches 43.4 and 33.8, respectively.Under Qwen, orchestration slightly trades off Multi-Hop and Temporal performance relative to +Long-Term alone.Source: §4.3 · p. 8, Table 5 · p. 8
4.4 Further Analysis
- MMKG-only consistently improves on Flat RAG across task categories, with particularly large gains in Multi-Hop and Temporal reasoning.This comparison uses GPT-4o-mini.Source: §4.4 · p. 8, Table 6 · p. 8
- Memory extraction takes 0.20s per 1k tokens, while parametric memory training takes 2.16s per 1k training tokens.The authors attribute longer total training time primarily to multiple epochs and large contextual training data.Source: §4.4 · p. 8, Table 7 · p. 8
- Further analyses examine periodic-update knowledge retention on ScienceQA and scale the backbone from Qwen2.5-1.5B to 72B with the Qwen2.5-7B-based memory module fixed.The passage points to Appendices E and F for details but does not report those analyses' outcomes.Source: §4.4 · p. 8
5 Conclusion
MemVerse combines fast parametric recall with slow hierarchical retrieval to organize multimodal experiences and support long-horizon agent memory. The conclusion reports improvements across multimodal tasks and identifies adaptive memory control and open-world deployment as future directions.
5 Conclusion
- MemVerse unifies fast parametric recall with slow hierarchical retrieval, transforming multimodal experiences into structured knowledge while avoiding unbounded memory growth.The framework also maintains recent context and retrieves information from its memory.Source: §5 · p. 8
- Across diverse multimodal tasks, MemVerse yields improvements in reasoning accuracy, stability, and long-horizon coherence.Source: §5 · p. 8
- Future work will explore more adaptive memory-control strategies and deploy MemVerse in open-world environments across varied domains.Source: §5 · p. 9
6 Limitations
MemVerse’s graph-based memory has been evaluated primarily with small- to medium-scale memories. Longer interactions may increase retrieval and insertion overhead and introduce noise from outdated memories.
6 Limitations
- MemVerse’s graph-based memory organization has so far been evaluated primarily under small- to medium-scale memory settings.Source: §6 · p. 9
- In longer-term interactions, graph expansion may increase retrieval and insertion overhead and potentially introduce noise from outdated memories.Source: §6 · p. 9
- The authors propose partitioning memories into temporal graphs and using temporal indexing to access archived graphs while preserving retrieval efficiency.Source: §6 · p. 9
A Datasets
The datasets cover long-term conversational memory, multimodal science question answering, and video–text retrieval. They provide varied tests of memory, visual-textual reasoning, and video alignment.
A Datasets
- LoCoMo contains 10 multimodal dialogues of up to 32 sessions each, averaging 600 turns and 16,618 tokens per conversation.It tests temporal reasoning, persona consistency, and multimodal grounding over extended interactions.Source: A Datasets · p. 14
- LongMemEval evaluates conversational assistants’ ability to store, retrieve, update, and reason over information across long, multi-session chat histories.Its results show significant long-term memory performance degradation for state-of-the-art long-context LLMs and commercial assistants.Source: A Datasets · p. 14
- ScienceQA contains 21,208 science questions, with images in approximately 48.7% of instances, requiring cross-modal reasoning between text and visual content.Source: A Datasets · p. 14
- MSR-VTT pairs 10,000 YouTube clips across 20 categories with 200,000 human-annotated captions for video–text retrieval and related tasks.Source: A Datasets · p. 14
B Implementation Details
The implementation evaluates MemVerse against language, vision-language, and memory-augmented baselines across the paper’s benchmarks. It textualizes multimodal inputs, extracts three memory types, and constructs separate knowledge graphs for memory instances.
B.1 Baselines
- Baselines span proprietary and open-source language models, memory frameworks, multimodal reasoning systems, and video–text retrieval methods.Experiments use identical prompting and evaluation protocols; the MSR-VTT baseline uses only a CLIP text encoder.Source: §B.1 · pp. 15–16
B.2 Multimodal Preprocessing
- Images are captioned, audio is transcribed, and video frames are captioned before each resulting text segment is treated as an independent chunk.The pipeline uses GPT-4o-mini for image captions and Whisper for audio transcription.Source: §B.2 · p. 16
B.3 Memory Extraction Prompts
- A unified prompting pipeline converts textualized multimodal chunks and original text into core, semantic, and episodic memories for downstream retrieval and reasoning.Source: §B.3 · p. 16
B.4 Knowledge Graph Construction
- LLM-based entity and relation extraction builds one structured multimodal knowledge graph per memory instance, keeping graphs relatively small for efficient insertion and retrieval.Source: §B.4 · p. 16
B.5 Training Details
MemVerse training builds parameterized memory from multimodal datasets and uses causal language modeling to train Qwen2.5-7B. Its algorithms describe online inference and memory construction with parametric distillation.
B.5 Training Details
- MMKGs are built from ScienceQA training data, LoCoMo session dialogues, and MSR-VTT without labels; Qwen2.5-7B is fine-tuned to encode their distilled multimodal knowledge.Training uses causal language modeling on Question-Retrieved pairs, with loss applied only to the output segment.Source: §B.5 · p. 16
C Algorithms
- Algorithm 1 takes a query, short-term memory, three long-term graphs, and parametric memory, and returns a response and memory context.The algorithms section distinguishes online inference in Algorithm 1 from memory update and construction in Algorithm 2.Source: §B.5 · p. 17, C Algorithms · p. 17
- Algorithm 2 extracts and categorizes memory entries, parses entities and relations, merges entities, updates graphs, and constructs training pairs from long-term memory for distillation.Its stated inputs include conversation history, long-term memory, an orchestrator, and parametric memory; its outputs are updated memory structures and parametric memory.Source: C Algorithms · p. 17
D Additional Results
MemVerse-enhanced models achieve top ScienceQA scores across several categories, and memory-augmented retrieval substantially outperforms CLIP on MSR-VTT. The video-retrieval setup uses linked caption representations without exposing ground-truth caption–video alignments.
ScienceQA
- 85.48% is the highest average ScienceQA score, achieved by GPT-4o-mini with MemVerse; it also leads in natural science (85.26%), social science (81.55%), and language (89.09%).MemVerse also leads for text context (83.28%) and image caption (78.19%).Source: D Additional Results · p. 17, Table 8 · p. 18
Memory construction
- Algorithm 2 extracts and categorizes memory entries, parses and merges entities, updates graphs, and constructs training pairs from long-term memory for parametric distillation.The algorithm takes conversation history, long-term memory, an orchestrator, and parametric memory, and outputs updated memory structures and parametric memory.Source: D Additional Results · p. 17
MSR-VTT
- R@1 reaches 90.4% for text-to-video and 89.2% for video-to-text, exceeding CLIP by 60.7% and 67.8% points, respectively.At inference, query text retrieves memory-graph entries for query rewriting before final matching.Source: D Additional Results · pp. 17–18
- Caption pairs are partially aligned through LLM judgment and reasoning, then stored as linked representations; ground-truth caption–video alignment is not exposed to the graph or parametric module.This combines embedding-based retrieval scalability with semantic information from large pretrained models.Source: D Additional Results · p. 18
E Memory Evolution Performance
The memory-evolution experiments simulate staged fine-tuning on progressively larger portions of retrieved knowledge. The best reported overall accuracy occurs at the 75% update stage, while the benefit of additional memory depends on model capacity and dataset characteristics.
E Memory Evolution Performance
- The experiment compares the original Qwen2.5-7B model with staged fine-tuning on the earliest 25%, 50%, 75%, or 100% of retrieved-memory segments.This simulates incremental memory growth and examines retention, stability, and generalization under temporally ordered updates.Source: E Memory Evolution Performance · p. 18
- With 25% of accumulated knowledge, recall strengthens in NO and IMG; at 50%, gains broaden across subject areas, particularly SOC and NAT.The 50% stage provides more semantically diverse supervision, which the authors say enhances learned-memory stability.Source: E Memory Evolution Performance · pp. 18–19
- 75.69 is the highest overall accuracy, reached at the 75% update stage; performance plateaus and slightly degrades at 100%.The authors attribute the limited extra benefit to ScienceQA’s broad scientific knowledge and the base model’s strength; smaller models such as 1.5B show more noticeable improvement with more memory.Source: E Memory Evolution Performance · p. 19
F Scalability Analysis of MemVerse
The section reports parametric-memory gains across Qwen2.5 model scales, reviews related memory and multimodal retrieval work, and includes an example memory-retrieval case.
F Scalability Analysis of MemVerse
- Table 10 frames staged fine-tuning as incremental memory growth and identifies 50%–75% update intervals as the most stable and effective consolidation.The caption describes comparison of the original Qwen2.5-7B baseline with progressively larger retrieved-knowledge partitions.Source: Table 10 · p. 20
- +7.61 points at 1.5B, +0.90 at 7B, +1.24 at 14B, +0.92 at 32B, and +1.88 at 72B are the reported overall gains with parametric memory.The comparison covers base models and their parametric-memory-enhanced variants across ScienceQA categories, modalities, and grade levels.Source: F Scalability Analysis of MemVerse · pp. 19–20, Table 11 · p. 20
G.1 Memory for LLM Agents.
- Related work covers parametric adaptation and external retrieval; MemVerse presents a CLS-based fast–slow framework for efficient recall and stable memory consolidation.The related-work discussion characterizes existing retrieval-based memory as interaction-inefficient and purely parametric memory as lacking reliable long-term retention.Source: §G.1 · pp. 20–21
G.2 Multimodal Knowledge Retrieval
- MemVerse organizes multimodal experiences into MMKGs to address flat retrieval’s limits for complex semantics and hidden cross-modal relations, recasting retrieval as unified text-based retrieval.The passage says this transformation is made without information loss.Source: §G.2 · p. 21
H Example Cases
- Figure 6 illustrates a case in which a stored memory is queried and retrieved.Source: H Example Cases · p. 21, Figure 6 · p. 22