PaperlayerView exact revision on arXiv

Source-linked AI summary

EnSIMem: Entity-Structured Indexing for Long-Term Agent Memory

Xuanyu Meng, Xing Fan, Xinyi Fan, Chenlei Guo, Yixuan Xie, Jiawei Han

arXiv:2609.27279v2cs.AIcs.CL

TL;DR

Long-term agents need to recover the right facts and evidence from growing interaction histories, while summaries and anonymous chunks may omit important details or entity-property links. EnSIMem builds theme-coherent episodes and dialogue-grounded entity-property indexes, then retrieves source evidence according to each query's requirements. On long-term conversational-memory benchmarks, it reports high answer accuracy with compact, evidence-focused contexts.

  • Problem

    Long-term agents must retrieve relevant facts and evidence from growing interaction histories, while full histories expand context and summaries or anonymous chunks can lose details, provenance, or entity-property specificity.

    Source: §1 · pp. 1–2

  • Method

    EnSIMem organizes interactions into theme-coherent episodes, indexes dialogue-grounded entity-property records linked to source turns, and adaptively retrieves evidence for decomposed query requirements.

    Source: §1 · p. 2

  • Results

    EnSIMem achieves high answer accuracy with compact, evidence-focused retrieval contexts on long-term conversational-memory benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EnSIMem's evidence-preserving design provides a foundation for reliable long-term memory and future extensions to semantic and procedural memory.

    Source: §6 · p. 9

    Online latency is higher than MEMORA, with most added cost from requirement-aware planning and evidence retrieval.

    Source: §5 · p. 9

Abstract

from arXiv · show

An agent that interacts with users over long periods must recall facts, preferences, events, and changes from a continuously growing interaction history. Existing memory systems often compress interactions into generic summaries or retrieve anonymous text chunks, making it difficult for an agent to identify the correct entity, property, and supporting evidence. We present EnSIMem, an entity-structured long-term memory architecture for an agent. During offline construction, the system organizes interactions into theme-coherent episodes and builds dialogue-grounded index entries of the form [entity][entity_type][property: value]. Each entry preserves its source turns, temporal information, and available multimodal fields. During online interaction, the agent's request is decomposed into evidence requirements whose properties are aligned with the memory index. Entity-property lookup and adaptive retrieval then collect the evidence needed for point, temporal, compositional, and aggregation reasoning. The agent generates its response from the preserved source evidence rather than from lossy memory summaries. On long-term agent-memory benchmarks, EnSIMem achieves high answer accuracy while maintaining compact, evidence-focused contexts. These results show that entity-structured indexing and episode-level provenance provide a reliable foundation for long-term memory in agents. The code of our model is available at https://github.com/RamonMeng/EnSIMem.

1 INTRODUCTION

Long-term agents must recover relevant evidence across interactions expressed in different ways, while avoiding the context growth of full histories and the detail and provenance loss of summaries. EnSIMem uses entity-property structure to locate evidence and preserved dialogue to interpret it.

1 INTRODUCTION

  • Long-term memory must connect differently worded references to the same event without erasing event-specific distinctions or losing source evidence.Full histories increase context growth, latency, and distraction; summaries can remove details, temporal qualifiers, and provenance.Source: §1 · pp. 1–2
  • Responses are generated from retrieved source evidence, allowing answers to be traced from query requirements through index access to supporting dialogue and multimodal observations.The index identifies where to look; the original episode supplies context for interpretation and answer synthesis.Source: §1 · p. 2
  • EnSIMem achieves high answer accuracy with compact, evidence-focused retrieval contexts on long-term conversational-memory benchmarks.The authors present theme-coherent episodes, dialogue-grounded indexing, and requirement-aware retrieval as a practical foundation for reliable long-term agent memory.Source: §1 · p. 2
  • Offline construction organizes interactions into theme-coherent episodes and indexes entities and salient events in records linked to original dialogue and multimodal observations.Records use an intermediate property granularity: broad enough to connect paraphrases but specific enough to preserve the underlying relation or event.Source: §1 · p. 2
  • Online retrieval decomposes requests into evidence requirements and uses structured entity-property matching with adaptive budgets for point, temporal, compositional, and aggregation queries.Property alignment supports paraphrase matching without erasing event-specific distinctions.Source: §1 · pp. 2–3
  • The evidence-rich reasoning buffer is compact and query-specific, retaining relevant source evidence while filtering unrelated noise without additional training or supervision.The introduction describes the buffer as the basis for the reported benchmark results.Source: §1 · p. 3

2 RELATED WORK

Related work spans episodic, semantic, and procedural agent memory, as well as retrieval and evaluation systems for long conversations and long documents. EnSIMem is positioned as episodic conversational memory that uses structure to address preserved evidence rather than replace it.

Memory in long-running agents

  • Episodic, semantic, and procedural memory serve different functions and impose different update and evaluation requirements; EnSIMem is confined to episodic conversational memory.Extending its address-and-provenance principle to semantic or procedural memory remains future work.Source: §2 · p. 3

Recent agent-memory systems and evaluations

  • Recent agent-memory work treats memory as an agent capability, with systems for memory graphs, learned memory operations, experience distillation, evolving user state, and reusable skills.Recent evaluations also include multimodal conversations and hidden, evolving user state.Source: §2 · p. 4

Entity-structured retrieval for long documents

  • Long-document retrieval and context-selection systems study dense, graph-aware, and structured access, but do not target persistent dialogue-grounded memory with temporal and multimodal provenance.EnSIMem extends entity-structure indexing to agent memory by linking records to episodes and query-aware evidence budgets.Source: §2 · p. 4

Structured agent memory

  • Structured agent-memory baselines improve compactness or navigation but may lose local wording, provenance, or a direct address to the requested entity-property pair.EnSIMem instead uses structure as an address into preserved episodes so the answer model can reason over original turns.Source: §2 · p. 4

3 ENSIMEM

EnSIMem separates compact memory navigation from answer evidence: it organizes dialogue into episodes, indexes supported entity properties, and retrieves source turns according to each query’s evidence needs. The answer is generated from restored episode evidence, preserving context for temporal, compositional, and aggregative reasoning.

3.1 SYSTEM OBJECTIVE AND DESIGN PRINCIPLE

  • EnSIMem uses structured records to locate relevant evidence, then restores original interactions so answers rely on source context rather than compressed memories.This separates localization from interpretation: the index is an address system, not a substitute for the interaction.Source: §3.1 · p. 4

3.2 OFFLINE EPISODIC MEMORY CONSTRUCTION

  • Episodes group contiguous turns around the same entity, event, goal, or unresolved reference, while linked records capture explicitly mentioned entities and supported properties.Intermediate property normalization aligns paraphrases while retaining distinctions, and episodes preserve context needed to interpret records.Source: §3.2 · pp. 4–5

3.3 ONLINE REQUIREMENT-AWARE RETRIEVAL

  • For each request, a planner specifies atomic evidence requirements and reasoning type; structured matching and dense fallback retrieve episodes until those requirements are covered.The stopping rule seeks the smallest evidence set sufficient for the requested operation, rather than relying on similarity or a fixed top-k.Source: §3.3 · p. 5

3.4 EVIDENCE-GROUNDED RESPONSE AND WHY IT WORKS

  • Selected episodes are expanded into a compact buffer of original turns and metadata, and the answer is generated from that evidence rather than index records alone.The planner and evidence buffer support different needs, such as retaining an event’s source time or covering multiple values in an aggregation.Source: §3.4 · p. 6

4 EXPERIMENTS

EnSIMem achieves high answer accuracy on LoCoMo and LongMemEval, while its online efficiency and ablations examine the costs and effects of its design choices. The ablations favor theme-coherent episodes, intermediate property normalization, structured matching, and adaptive retrieval.

4.1 EVALUATION ON THE LOCOMO BENCHMARK DATASET

  • 90.6% overall accuracy with GPT-4o-mini and 90.0% with Qwen3-32B; EnSIMem is strongest on single-hop and temporal questions, while open-domain questions remain hardest.The GPT-4o-mini result is the directly comparable result against published MEMORA numbers.Source: §4.1 · p. 6

4.2 EVALUATION ON THE LONGMEMEVAL BENCHMARK DATASET

  • Both evaluation models reach 92.8% overall on LongMemEval, with the largest gains over MEMORA (P) on SS-assistant and multi-session.The two evaluation-model results are reported separately rather than averaged.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 EFFICIENCY

  • EnSIMem takes 14.4 seconds online versus MEMORA’s 5.70 seconds, while using 7,470 context tokens versus 8,499 under potentially different accounting conventions.Requirement-aware planning takes 10.3 seconds and retrieval 1.59 seconds; the token comparison is directional.Source: §4.3 · p. 7, Table 3 · p. 8

4.4 ABLATION STUDY

  • 96.59% accuracy with theme-coherent episodes exceeds 94.32% for per-session and 90.91% for per-turn units.The comparison is shown in Figure 2; the prose attributes the difference to memory-unit granularity.Source: §4.4 · p. 7, Figure 2 · p. 8
  • 96.59% accuracy with Broad properties exceeds 92.05% for Fine and 90.91% for Very broad, while Broad+ reaches 95.45%.The results indicate that moderate normalization helps, while excessive coarsening removes distinctions needed for retrieval.Source: §4.4 · p. 8, Figure 2 · p. 8
  • 96.59% accuracy with structured matching exceeds 86.36% for dense retrieval when answer and evaluation models are fixed.Structured matching uses entity, type, and property fields and links records to source episodes.Source: §4.4 · p. 8
  • 96.59% accuracy with adaptive retrieval exceeds fixed top-k retrieval, which rises from 52.27% at k = 1 to 95.45% at k = 8 before saturating or declining.Figure 3 presents the comparison; adaptive allocation is query-type-aware.Source: §4.4 · p. 8

5 DISCUSSION, LIMITATIONS, AND FUTURE WORK

The study focuses on episodic conversational memory, with online latency as its main practical limitation. It leaves semantic and procedural extensions for future work.

5 DISCUSSION, LIMITATIONS, AND FUTURE WORK

  • The evaluation scope is episodic conversational recall, not procedural skill execution or long-term semantic consolidation.The pipeline also relies on LLM-based segmentation, property extraction, and query planning, so changes in underlying models may affect the intermediate representation.Source: §5 · p. 8
  • Requirement-aware planning and evidence retrieval account for most of EnSIMem’s additional online latency relative to MEMORA.The authors suggest caching recurring plans, batching property matching, distilling planning, and early stopping as possible optimizations.Source: §5 · p. 9
  • Semantic and procedural memory extensions remain future work, including their concrete representations, algorithms, and evaluation.The paper gives stable facts and reusable action-related information as examples of what those memory types would extract while preserving evidence links.Source: §5 · p. 9

6 CONCLUSION

EnSIMem uses structured records as handles for retrieving original dialogue evidence, turning long-term conversational recall into short-context reasoning. The conclusion presents this evidence-preserving design as a foundation for reliable agent memory and possible future extensions.

6 CONCLUSION

  • Theme-coherent episodes and entity-property records locate evidence while preserving the local context needed to interpret references.The records provide precise handles, while the reasoning buffer contains original source text, provenance, and available multimodal evidence rather than generalized summaries.Source: §6 · p. 9
  • Requirement-aware planning and adaptive retrieval select evidence for point, temporal, compositional, and aggregation questions.Source: §6 · p. 9
  • The authors present the evidence-preserving design as a foundation for reliable long-term memory and future semantic and procedural memory extensions.Source: §6 · p. 9

AI USE DISCLOSURE

The authors used generative AI tools for writing support and software implementation, and used specified models in benchmark experiments. They report that AI tools were not used to generate synthetic datasets or establish unverified proofs or mathematical claims.

AI USE DISCLOSURE

  • Generative AI tools were used to polish writing, check grammar, and implement and debug software.Source: AI USE DISCLOSURE · p. 9
  • GPT-4.1-mini generated benchmark answers, while GPT-4o-mini and Qwen3-32B evaluated them; these evaluations did not create synthetic training data or alter ground-truth annotations.Source: AI USE DISCLOSURE · p. 9
  • The authors report that AI tools did not generate synthetic datasets or provide proofs or mathematical claims without author verification, and that they reviewed AI-assisted code and manuscript materials.Source: AI USE DISCLOSURE · p. 10

REPRODUCIBILITY STATEMENT

The supplementary packages include scripts, prompts, configurations, and instructions for reproducing LoCoMo and LongMemEval experiments. Public benchmark data are obtained from official releases and are not redistributed.

REPRODUCIBILITY STATEMENT

  • The LoCoMo package covers preprocessing, episode and index construction, retrieval, answer generation, ablations, and evaluation; LongMemEval includes benchmark preparation, inference, and evaluation.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Both packages include prompts, configuration files, input/output formats, result aggregation scripts, and execution instructions; benchmark data are not redistributed and API credentials are excluded.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Experiments use GPT-4.1-mini for answer generation, GPT-4o-mini and Qwen3-32B as evaluation models, temperature 0, and random seed 42 where supported.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 BENCHMARK OVERVIEW

Table 5 covers benchmark scope, question families, and evaluation protocol.

A.1 BENCHMARK OVERVIEW

  • Table 5 outlines benchmark scope, question families, and evaluation protocol.Source: Table 5 · p. 15

A.2 BASELINE AND METHOD OVERVIEW

The section presents a baseline and method overview, then illustrates three distinct EnSIMem case-study mechanisms: cross-episode evidence coverage, temporal grounding, and fine-grained entity-property localization.

A.2 BASELINE AND METHOD OVERVIEW

  • Table 6 presents the baseline and method comparison used in the benchmark tables.Source: Table 6 · p. 16

B CASE STUDIES

  • The three case studies examine cross-episode evidence coverage, temporal grounding, and fine-grained entity-property localization.Each table records the question, required operation, memory representation, retrieved evidence, generated responses, and the competing system’s failure mode.Source: B CASE STUDIES · p. 17
  • Case study 1 focuses on cross-episode evidence coverage for a multi-fact question.Source: Table 7 · p. 17
  • Case study 2 focuses on temporal grounding from a relative-time expression.Source: Table 8 · p. 18, B CASE STUDIES · p. 18
  • Case study 3 focuses on entity-property specificity for multimodal art evidence.Source: Table 9 · p. 19

C EVALUATION MODEL RATIONALE

The evaluation rationale favors answer-level semantic assessment for open-ended responses, while using separate task-specific checkers for LoCoMo and LongMemEval.

C EVALUATION MODEL RATIONALE

  • LoCoMo maps binary CORRECT/WRONG judgments to scores and excludes Category 5 under the Memora-compatible policy.Source: C EVALUATION MODEL RATIONALE · p. 20
  • LongMemEval uses question-type-specific checks for preferences, knowledge updates, temporal tolerance, and unanswerable items.Source: C EVALUATION MODEL RATIONALE · p. 20
  • The protocol evaluates semantic correctness rather than lexical overlap, which can penalize valid paraphrases and equivalent date formats.The evaluator receives the question, gold answer, and generated response.Source: C EVALUATION MODEL RATIONALE · p. 20
  • The evaluation models are reported separately because model-based evaluation may reflect evaluator-specific preferences.The protocols use fixed prompts and temperature zero; LoCoMo also uses seed 42.Source: C EVALUATION MODEL RATIONALE · p. 20
  • Table 10 describes score consistency between GPT-4o-mini and Qwen3-32B, including percentage-point differences and category-level correlation.Source: Table 10 · p. 21

D EVALUATION-MODEL CONSISTENCY

The evaluation-model consistency analysis finds similar aggregate conclusions across two evaluators, while distinguishing aggregate stability from per-query agreement. The appendix also specifies the memory-construction, retrieval, response-generation, and evaluation prompts.

D EVALUATION-MODEL CONSISTENCY

  • The analysis compares unrounded category-level results without treating the evaluators’ arithmetic mean as a new score.Source: D EVALUATION-MODEL CONSISTENCY · p. 21
  • The LongMemEval overall gap is 0.030 pp (92.8% versus 92.8%), while the LoCoMo gap is 0.610 pp (90.6% versus 90.0%).Category-level MAE is 1.37 pp on LongMemEval and 2.11 pp on LoCoMo; correlations are r = 0.936 and r = 0.946, respectively.Source: D EVALUATION-MODEL CONSISTENCY · p. 21
  • The stability analysis does not establish perfect per-query agreement; paired reliability measures would require both evaluators’ labels for each identical query.Source: D EVALUATION-MODEL CONSISTENCY · p. 21

E.1 OFFLINE MEMORY CONSTRUCTION

  • Offline construction partitions dialogue into contiguous theme-coherent episodes, then extracts query-independent entity-property records grounded in source episodes.Episodes preserve dialogue order and attach image metadata to the turn that introduced it.Source: §E.1 · p. 22

E.2 ONLINE RETRIEVAL AND RESPONSE GENERATION

  • Online planning converts questions into explicit evidence requirements and a hop graph aligned with the index schema.Source: §E.2 · p. 23
  • The answer model receives complete retrieved episodes and must ground its response in the original dialogue and attached image metadata.Source: §E.2 · p. 24

E.3 EVALUATION PROMPTS

  • Benchmark evaluation is separate from answer generation and uses binary judgments, with LongMemEval selecting task-specific prompts.The appendix describes prompt templates and runtime substitutions for the LLM stages.Source: E PROMPTS · p. 22, §E.3 · pp. 24–25
Loading 2609.27279v2…