PaperlayerView exact revision on arXiv

Source-linked AI summary

TeleMem: Building Long-Term and Multimodal Memory for Agentic AI

Chunliang Chen, Ming Guan, Xiao Lin, Jiaxu Li, Luxi Lin, Qiyi Wang, Xiangyu Chen, Jixiang Luo, Changzhi Sun, Dell Zhang, Xuelong Li

arXiv:2601.06037v4cs.CLcs.AIcs.CV

TL;DR

Long-term agents struggle because finite context and unordered retrieval fail to preserve evolving, causally dependent information. TELEMEM combines narrative-grounded multimodal memory, structured writing, threaded DAG retrieval, and ReAct-style reasoning. It reports higher accuracy, lower token use, and faster operation than Mem0 on ZH-4O, while bounded traversal remains necessary when memory graphs grow large.

  • Problem

    Finite context and flat retrieval make it difficult to preserve temporal order, causal dependency, evolving states, and reliable long-term multimodal memory.

    Source: §2 · p. 1

  • Method

    TELEMEM uses narrative-grounded extraction, batched clustering and consolidation, threaded DAG memory with closure-based retrieval, and ReAct-style multimodal reasoning.

    Source: §2 · p. 2

  • Results

    TELEMEM outperforms Mem0 by 19% in accuracy, uses 43% fewer tokens, and achieves a 2.1× speedup on ZH-4O.

    Source: §2 · p. 2

  • Takeaways & Limitations

    TELEMEM provides coherent user profiles, more efficient memory operations, and closed-loop reasoning over complex video content.

    Source: §2 · p. 2, §6 · p. 10

    Closure-based traversal may require maximum-depth bounds or lightweight relevance and budget heuristics as graphs grow, to control latency and context size.

    Source: §3.3.1 · p. 6

Abstract

from arXiv · show

Large language models (LLMs) excel at many NLP tasks but struggle to sustain long-term interactions due to limited attention over extended dialogue histories. Retrieval-augmented generation (RAG) mitigates this issue but lacks reliable mechanisms for updating or refining stored memories, leading to schema-driven hallucinations, inefficient write operations, and minimal support for multimodal reasoning.To address these challenges, we propose TeleMem, a unified long-term and multimodal memory system that maintains coherent user profiles through narrative dynamic extraction, ensuring that only dialogue-grounded information is preserved. TeleMem further introduces a structured writing pipeline that batches, retrieves, clusters, and consolidates memory entries, substantially improving storage efficiency, reducing token usage, and accelerating memory operations. Additionally, a multimodal memory module combined with ReAct-style reasoning equips the system with a closed-loop observe, think, and act process that enables accurate understanding of complex video content in long-term contexts. Experimental results show that TeleMem surpasses the state-of-the-art Mem0 baseline with 19% higher accuracy, 43% fewer tokens, and a 2.1x speedup on the ZH-4O long-term role-play gaming benchmark.

2. Introduction

Long-horizon agents need memory that preserves temporal order, causal dependency, and evolving states, because finite context and flat retrieval leave distant information fragmented. TELEMEM addresses these gaps with narrative-grounded extraction, structured memory organization, efficient writing, and multimodal reasoning.

  • Finite context windows degrade recall of distant user-specific facts and destabilize long-horizon reasoning, limiting persistent personalization and continuous learning.Source: §2 · p. 1
  • Long-term memory must preserve semantic similarity alongside temporal order, causal dependency, and state evolution to avoid fragmented contexts and missing prerequisites.Source: §2 · p. 1
  • Flat RAG stores unordered fragments and lacks reliable updating, consolidation, and deletion, making evolving states and causal consistency difficult to preserve.Source: §2 · p. 1
  • TELEMEM extracts dialogue-supported narrative units and organizes semantic trajectories to maintain compact, coherent, hallucination-resistant user profiles.Source: §2 · p. 2
  • Its structured writing pipeline batches retrieval, clustering, and consolidation, reducing storage redundancy, token overhead, and end-to-end latency.Source: §2 · p. 2
  • TELEMEM adds a multimodal memory module with ReAct-style observe–think–act reasoning for complex visual content.Source: §2 · p. 2
  • 19% higher accuracy, 43% lower token usage, and a 2.1× speedup over Mem0 are reported on ZH-4O.Source: §2 · p. 2

3. TELEMEM System

TELEMEM threads long-term memory into a directed acyclic graph rather than isolated vector chunks, coupling representation, graph organization, and dependency-aware retrieval. This design reconstructs coherent causal context while supporting consolidation, updates, scalable growth, and low-latency access.

  • TELEMEM represents long-term memory as a directed acyclic graph with explicit relations, replacing unordered embeddings with traceable, evolvable memory threads.Source: §3 · p. 2
  • The architecture couples semantic representation and consolidation, persistent graph organization, and closure-based retrieval that reconstructs coherent causal context.Source: §3 · p. 2
  • This unified design supports scalable memory growth while preserving structural consistency and low-latency access.Source: §3 · p. 2

3.1. Memory Graph Formulation

TELEMEM formalizes memory as a temporally ordered DAG whose nodes hold stabilized semantic states and whose edges encode minimal causal dependencies. Root-to-node threads and closure-based retrieval preserve compact, interpretable causal context across heterogeneous textual and multimodal memories.

3.1. Memory Graph Formulation

  • Memory is formalized as a directed acyclic graph G = (V, E), with nodes V and directed edges E.Each node has an effective timestamp reflecting semantic validity order.Source: §3.1 · p. 2
  • Effective timestamps impose strict temporal ordering on edges, guaranteeing acyclic evolution.Source: §3.1 · pp. 2–3
  • A virtual root node anchors memories without explicit predecessors, preventing isolated fragments and maintaining global connectivity.Source: §3.1 · p. 3

3.1.1. The Node: A Unified Semantic Container

  • Nodes unify profile, event, entity, and object states, each storing consolidated content, an embedding, and an effective timestamp across modalities.Source: §3.1.1 · p. 3

3.1.2. The Edge: Threaded Causal Skeleton

  • Directed edges encode semantic conditioning from prior states, while temporal constraints enforce monotonic evolution and prevent circular dependencies.Source: §3.1.2 · p. 3
  • The graph retains only irreducible dependencies, pruning edges whenever an alternative directed path makes them transitive or redundant.Source: §3.1.2 · p. 3
  • Memory threads are root-to-node paths ordered by increasing effective timestamps, tracing how topics, states, or beliefs evolve over time.Source: §3.1.2 · pp. 3–4
  • Minimal causal threads remain compact and interpretable, supporting traceable reasoning, dependency-aware retrieval, and stable long-term knowledge accumulation.Source: §3.1.2 · p. 4

3.2. Memory Graph Updating

TeleMem updates a temporal memory graph through structured offline consolidation and bounded online repairs. Its writing pipeline consolidates memories, while graph operators preserve temporal validity, sparse dependencies, and low-latency maintenance.

3.2.1. Graph Update Operators

  • Insert retrieves Top-K historical candidates, prunes redundant parents, and materializes irreducible incoming edges without modifying existing nodes or edges.This preserves the global DAG property while restricting parent selection to a local semantic neighborhood.Source: §3.2.1 · p. 4
  • ReInsert recomputes a node’s incoming edges when its representation or dependencies change, providing local repair while preserving threaded semantics and the global DAG property.Source: §3.2.1 · p. 4

3.2.2. Offline Batch Updates

  • The writing pipeline summarizes batched interactions, retrieves related memories, clusters aligned entries, and uses LLM consolidation to produce stabilized updates.Consolidation resolves redundancy and inconsistency across new and historical content, assigning add, delete, update, or no-op actions.Source: §3.2.2 · pp. 4–5
  • Offline graph rebuilding processes nodes in timestamp order and parallelizes independent insertions, avoiding quadratic all-pairs comparison while preserving temporal validity and pruning.Source: §3.2.2 · p. 5

3.2.3. Online Incremental Updates

  • Online updates operate on single interactions without global clustering, deciding whether to add, update, or no-op after summarization and retrieval alignment.Node and graph-index updates are separated, and child nodes are refreshed when upstream nodes change or are deleted.Source: §3.2.3 · pp. 5–6
  • Periodic offline rebuilding re-optimizes topology to amortize local approximation errors and preserve long-term structural consistency.Source: §3.2.3 · p. 6

3.3. Memory Reading

TeleMem reads memory by reconstructing minimal closed subgraphs that retain relevant evidence and historical prerequisites. For multimodal queries, a ReAct-style agent iteratively selects retrieval and video tools, observes their outputs, and refines grounded responses.

3.3.1. Retrieval Paradigm: Minimal Closed Subgraph

  • Closure-based retrieval replaces isolated Top-K fragment stitching with dependency-complete subgraphs, preserving prerequisite consistency and improving long-horizon reasoning stability.Source: §3.3 · p. 6, §3.3.1 · p. 6
  • The system seeds retrieval with Top-K embedding matches, traverses ancestor dependencies to the virtual root, and serializes the resulting closure by timestamp.Depth limits and relevance heuristics can bound latency and context size when closures grow too large.Source: §3.3.1 · p. 6

3.3.2. Reasoning Paradigm: ReAct-Style Multimodal Agent

  • For complex multimodal queries, a ReAct-style agent reasons about missing evidence and chooses among memory retrieval, textual RAG, and video question answering.Source: §3.3.2 · p. 6
  • Iterative think–act–observe cycles append retrieved memories, localized clips, or visual answers to the history, progressively refining grounded multimodal responses.Source: §3.3.2 · p. 6

4. Experiments

Experiments evaluate TELEMEM against long-context, retrieval, and memory baselines on Chinese and English long-dialogue benchmarks. TELEMEM achieves the strongest ZH-4O accuracy, while LoCoMo results show category-specific strengths and remaining multi-hop challenges.

4. Experiments

  • The evaluation uses QA Accuracy for ZH-4O and GPT-4o LLM-as-a-Judge assessment of factual accuracy, relevance, and completeness for LoCoMo.ZH-4O contains 28 sessions and 1,068 questions; LoCoMo contains 10 sessions and 1,540 question-answer pairs.Source: §4 · pp. 6–7

4.1. Results on ZH-4O

  • 86.33% QA Accuracy makes TELEMEM the best ZH-4O method, exceeding Long Context LLM at 84.92% and Memobase at 76.78%.Table 1 covers 1,068 probing questions; RAG reaches 62.45%.Source: §4.1 · p. 7, Table 1 · p. 7
  • Memory read–write scaling improves performance as either model grows, with joint scaling strongest and reader gains saturating when the reader outscales the writer.The results support coordinated read–write capacity rather than optimizing either component alone.Source: §4.1 · p. 8

4.2. Results on LoCoMo

  • Long Context LLM leads LoCoMo overall at 70.71%, while retrieval baselines perform poorly across reasoning categories.The full interaction history is particularly advantageous for temporal questions.Source: §4.2 · p. 8
  • TELEMEM reaches 64.53% on single-hop and 78.47% on temporal LoCoMo reasoning, approaching Long Context LLM performance in these categories.Its memory abstraction captures salient factual and temporal information, but multi-hop performance is weaker.Source: §4.2 · p. 8

5. Related Work

Related work develops long-term memory through text, graph, parametric, and multimodal architectures. These approaches differ in how they organize, internalize, retrieve, and reason over persistent information.

Text LTM

  • Text long-term memory spans virtual-memory systems, cognitive-inspired forgetting and prioritization, and modular agentic consolidation frameworks.Examples include MemGPT, MemOS, MemoryBank, LightMem, Nemori, Mem0, A-Mem, and MIRIX.Source: §5 · p. 8

Graph LTM

  • Graph memory systems organize information hierarchically and dynamically to improve reasoning, consistency, scalability, and dialogue coherence.Graph-based approaches range from temporal knowledge graphs to reasoning trees and trainable graph memories.Source: §5 · p. 8

Parametric LTM

  • Parametric memory internalizes knowledge or retrieval into model weights or persistent hidden states, offering unified reasoning but facing update-cost and alignment challenges.Methods include residual side networks, differentiable search, LoRA-based personalization, and direct factual editing.Source: §5 · p. 9

Multimodal LTM

  • Multimodal memory systems preserve and retrieve long-horizon audio-visual or visual-textual information through entity representations, temporal consolidation, hierarchical retrieval, or visual latents.The surveyed systems motivate multi-granular representations and adaptive retrieval for continuous multimodal streams.Source: §5 · p. 9

6. Conclusion

TELEMEM combines narrative-grounded extraction, structured memory writing, and ReAct-style multimodal reasoning into one long-term memory system. On ZH-4O, it outperforms Mem0 in accuracy, token efficiency, and speed.

6. Conclusion

  • TELEMEM combines narrative-grounded extraction, batching, clustering, consolidation, and ReAct-style multimodal reasoning.The system maintains coherent user profiles while supporting observe–think–act processing of complex video content.Source: §6 · p. 10
  • TELEMEM substantially outperforms Mem0 on ZH-4O across accuracy, efficiency, and speed.The conclusion reports these improvements without specifying the individual metric values.Source: §6 · p. 10

A. Example Memory Entries

TeleMem serializes distinct memory types in a unified schema, with entries designed to preserve semantics, provenance, and temporal context across text and video interactions.

Unified schema

  • The examples clarify a unified schema, field semantics, and provenance information rather than enumerating every possible memory attribute.The serialized entries support heterogeneous profile, event, and object states within the memory graph.Source: A. Example Memory Entries · p. 12

Text profile memory

Text event memory

Video event memory

  • Video event entries encode localized multimodal observations with captions, aligned timestamps, clip references, or auxiliary annotations grounded in visual evidence.Figure 5 illustrates this memory type.Source: A. Example Memory Entries · p. 12, Figure 5 · p. 12

Object memory

Loading 2601.06037v4…