Source-linked AI summary
Beyond Dialogue Time: Temporal Semantic Memory for Personalized LLM Agents
Miao Su, Yucan Guo, Zhongni Hou, Long Bai, Zixuan Li, Yufei Zhang, Guojun Yin, Wei Lin, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng
TL;DR
Existing LLM memory methods organize memories by dialogue time and isolated points, missing actual event timing and durative information. TSM builds and uses a semantic timeline with durative memories, and experiments show consistent gains over existing baselines, including up to 12.2% absolute accuracy improvement.
Problem
Existing memory methods suffer from temporal inaccuracy and fragmentation because they prioritize dialogue time and point-wise entries.
Method
TSM constructs a semantic timeline, consolidates continuous related information into durative memories, and retrieves them using the query’s temporal intent.
Results
TSM consistently outperforms strong baselines on LONGMEMEVAL and LOCOMO, especially for multi-session understanding and temporal reasoning.
Takeaways & Limitations
TSM’s results support semantic-time grounding and duration-aware consolidation for effective long-term memory in LLM agents.
Takeaways & Limitations
TSM uses fixed temporal granularity, such as monthly intervals, which may not be optimal across all application domains.
Abstract
from arXiv · showhide
Memory enables Large Language Model (LLM) agents to perceive, store, and use information from past dialogues, which is essential for personalization. However, existing methods fail to properly model the temporal dimension of memory in two aspects: 1) Temporal inaccuracy: memories are organized by dialogue time rather than their actual occurrence time; 2) Temporal fragmentation: existing methods focus on point-wise memory, losing durative information that captures persistent states and evolving patterns. To address these limitations, we propose Temporal Semantic Memory (TSM), a memory framework that models semantic time for point-wise memory and supports the construction and utilization of durative memory. During memory construction, it first builds a semantic timeline rather than a dialogue one. Then, it consolidates temporally continuous and semantically related information into a durative memory. During memory utilization, it incorporates the query's temporal intent on the semantic timeline, enabling the retrieval of temporally appropriate durative memories and providing time-valid, duration-consistent context to support response generation. Experiments on LongMemEval and LoCoMo show that TSM consistently outperforms existing methods and achieves up to 12.2% absolute improvement in accuracy, demonstrating the effectiveness of the proposed method.
1 Introduction
Existing LLM memory methods organize information around dialogue time and isolated point-wise entries, creating temporal inaccuracy and fragmentation. TSM addresses both issues by grounding memory in semantic time, consolidating durative memories, and retrieving temporally appropriate context; experiments show consistent gains over strong baselines.
- Motivation: Existing memory methods use structured dialogue histories, but they inadequately model temporal information.Their limitations concern both how memories are organized and how temporal relevance is assessed.
- Temporal limitations: Dialogue-time organization can misplace memories when discussed events occur in the past, future, or over an ongoing interval.For example, a May 28 conversation about a May 29 trip creates a mismatch between dialogue time and event time.
- Temporal limitations: Point-wise storage fragments temporally continuous information, obscuring persistent states and evolving patterns.The paper contrasts this with human memory, which uses time to order and link experiences across long-term memories.
- TSM: TSM builds a semantic timeline, links temporally continuous and semantically related mentions, and consolidates them into durative summaries.These summaries capture long-term states such as topics and personas.
- TSM: TSM retrieves memories using the query’s semantic temporal intent rather than dialogue-time recency or semantics-only similarity.Its hierarchical mechanism updates temporal facts online and periodically consolidates summaries for long-term consistency.
- Results: TSM consistently outperforms strong memory baselines on LONGMEMEVAL and LOCOMO, with largest gains in multi-session understanding and temporal reasoning.The experiments support semantic-time grounding and duration-aware consolidation as core components of the framework.
2 Related Work
Related memory systems increasingly use persistent stores and graphs, but they generally do not model memory evolution over time or fully exploit semantic time during retrieval. This leaves graph-based recall vulnerable to temporal misalignment.
- Agent memory: Agent memory stores factual information such as user profiles, dialogue history, and world facts for consistency and personalization.The related-work discussion situates TSM within broader long-term memory systems for LLM agents.
- Graph-based memory: Graph-structured memory emerges as agents accumulate relational insights over time.This motivates graph-based representations for organizing accumulated memory.
- Prior systems: Mem0g extracts and updates salient facts in a knowledge graph, while A-MEM links structured notes and updates existing memories as new information arrives.These systems provide dynamic storage and relational organization but are described as distinct from TSM’s temporal focus.
- Research gap: Many graph-based memory systems store extracted facts without modeling how memory evolves during a time period.Their retrieval stage therefore underuses semantic time encoded in the graph, resulting in temporally misaligned recall.
3 Methodology
TSM builds hierarchical memory around semantic time: it records episodic facts in a temporal knowledge graph, consolidates them into durative representations, and retrieves memories using query-time constraints. Online updates maintain the graph, while periodic consolidation refreshes higher-level summaries.
- Durative Memory Construction: TSM organizes episodic interactions into durative memories that capture enduring user patterns beyond isolated point events.Temporal segmentation and semantic abstraction produce hierarchical, temporally anchored memory for long-term storage and constraint-aware retrieval.
- Episodic Memory Construction: TSM constructs a temporal knowledge graph as episodic memory, recording temporally grounded facts and entity summaries for later access and consistency checking.The graph provides temporal localization and semantic-time access rather than serving directly as the final retrievable memory content.
- Durative Memory Construction: Temporal slices partition the graph into fixed intervals, with one month used as the default implementation granularity.Entities within each slice are clustered with a Gaussian Mixture Model into coherent semantic themes, which are summarized as topics with retrieval embeddings.
- Semantic-time Guided Memory Utilization: Memory utilization infers a query’s semantic time constraint, performs dense retrieval, and filters or reranks candidates using temporal validity before semantic similarity.The process combines topics, personas, raw dialogue, and temporally valid TKG evidence to produce contextually grounded retrieval.
- Hierarchical Memory Update: TSM uses a dual-stage update mechanism that incrementally maintains the episodic graph online and periodically consolidates summaries during higher-cost updates.New entities and temporally grounded relations are integrated incrementally, while threshold-triggered consolidation reorganizes mentions and refreshes topic and persona snapshots.
4 Experiments
Experiments evaluate TSM on two long-term memory benchmarks using category-wise accuracy, efficiency measures, and ablations. TSM generally outperforms memory baselines, while temporal retrieval and durative summaries each contribute to performance.
- Experimental Setup: TSM is evaluated on LONGMEMEVAL and LOCOMO, which test long-context memory abilities across information extraction, reasoning, temporal understanding, updates, abstention, and related categories.LONGMEMEVALS uses approximately 115k-token histories, while LOCOMO contains 1,986 questions across five categories.
- Main Results: TSM achieves 74.80% overall accuracy on LONGMEMEVAL_S with GPT-4o-mini, surpassing A-MEM’s 62.60%.It reports new state-of-the-art results on Temporal, Multi-Session, and Knowledge-Update questions.
- Main Results: +20.30% Multi-Session accuracy and +22.56% Temporal accuracy highlight TSM’s gains on tasks dependent on durable context and semantic time.The reported improvements are associated with maintaining long-term contextual information and retrieving or updating context across time.
- Main Results: On LOCOMO, TSM is the best memory-based method, reaching 71.23% with Qwen3-30B and 76.69% with GPT-4o-mini.The full-text baseline performs best with Qwen3 on LOCOMO because its shorter conversations can fit directly within the model context.
- Efficiency: TSM reaches 74.80% accuracy with 1.57/2.39-second P50/P95 recall latency on LONGMEMEVAL_S, despite not using the fewest tokens.The efficiency advantage is attributed to pre-built memory, temporal filtering, and local reranking during recall.
- Ablation Study: Removing temporal retrieval lowers overall accuracy from 74.8% to 72.8%, while removing summaries lowers it to 73.6%.Temporal removal most affects Temporal questions, whereas summary removal notably harms Single-Session-Preference and Temporal tasks; both components contribute overall.
5 Conclusions
TSM combines semantic-time grounding with duration-aware memory management to capture continuous user experiences and improve retrieval. Experiments on LONGMEMEVAL and LOCOMO show stronger QA accuracy, especially for multi-session understanding and temporal reasoning.
- TSM combines semantic-time grounding with duration-aware management to capture continuous user experiences beyond isolated point-wise entries.It consolidates temporally continuous and semantically related information into durative memory.
- TSM improves contextual consistency and retrieval by consolidating continuous information and incorporating temporal intent during memory utilization.
- TSM significantly outperforms existing memory baselines in QA accuracy, particularly on multi-session understanding and temporal reasoning tasks.
Limitations
The paper identifies fixed temporal granularity and a personalization-focused scope as limitations. Adaptive granularity and extensions to other memory paradigms remain future work.
- TSM uses fixed temporal granularity, such as monthly intervals, which may not suit every application domain.Adaptive selection based on event temporal density is left for future work.
- The work focuses on personalization applications rather than procedural memory for agent learning or shared memory for multi-agent systems.Extending TSM to these memory paradigms remains future work.
- Adaptive granularity selection based on event temporal density is identified as a future improvement for flexibility.
A.1.1 Parameter Setup
The appendix describes experimental and case-study settings, including temporal constraints, retrieval inputs, embedding choices, and example dialogue content. It also identifies Figure 3 and Table 5 as supporting materials for the setup.
- The example query asks about a cocktail recipe tried last weekend, with retrieved dialogue including a lavender gin fizz discussion.
- The case study uses a semantic time constraint spanning 2023/05/22–2023/05/28.
- TSM retrieves chat turns and applies temporal filtering before using the case-study context.
- The appendix organizes the case study around culinary applications, herbs and spices, healthy eating, and beverage pairings.
- The setup uses text-embedding-3-small and also evaluates text-embedding-v4 from Qwen.
A.1.2 Hardware
Experiments run on a machine with eight Nvidia A100 GPUs and 256 GB of total system memory.
- The experiments use a machine equipped with 8 Nvidia A100 GPUs.
- Each Nvidia A100 GPU has 80 GB of RAM.
- The machine provides 256 GB of total system memory.
A.2 Case Study
The case study evaluates TSM on long-term memory benchmarks and illustrates how temporal constraints and semantic retrieval jointly select relevant memories. It also specifies automated judging procedures for correctness, temporal reasoning, knowledge updates, preferences, and abstention.
- Case Study: Figure 3 applies a query’s temporal expression to retrieve facts within a semantic time range before dense retrieval over raw dialogue and summaries.For “last weekend” on 2023-05-30, the system derives 2023-05-22 to 2023-05-28 as the relevant interval.
- Case Study: TSM ranks the correct cocktail recipe first after combining temporal filtering with dense retrieval.
- Case Study: TKG facts alone cannot directly answer the example question because they provide structured, time-aware but insufficiently detailed point-wise knowledge.
- Evaluation: The evaluation prompts judge whether generated responses match correct answers, including equivalent time references and updated information where applicable.Separate prompt families cover correctness, temporal reasoning, knowledge updates, preferences, and abstention.