PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Memory Need Graphs? A Unified Framework and Empirical Analysis for Long-Term Dialog Memory

Sen Hu, Yuxiang Wei, Jiaxin Ran, Zhiyuan Yao, Xueran Han, Huacan Wang, Ronghao Chen, Lei Zou

arXiv:2601.01280v3cs.CLcs.AI

TL;DR

Dialog memory research lacks consistent evidence about which graph and non-graph design choices matter, partly because systems combine different foundational settings. The paper introduces a unified framework and controlled stage-wise analysis, finding that foundational settings substantially affect performance and that graph-based memory helps only under some configurations. It uses these findings to establish strong baselines for fairer future comparison and system development.

  • Problem

    Prior dialog memory studies report inconsistent results for graph-based and non-graph approaches, making specific design effects difficult to attribute.

  • Method

    The paper decomposes dialog memory systems into unified components and compares memory representation, organization, maintenance, indexing, and retrieval through controlled experiments.

  • Results

    Foundational system settings substantially affect performance, while graph-based memory helps under some configurations but can degrade results when graph construction or retrieval is inappropriate.

  • Takeaways & Limitations

    The framework and validated strong baselines provide a common starting point for fair comparison and practical dialog memory system development.

  • Takeaways & Limitations

    The framework does not cover all customized memory systems, and the experiments do not exhaustively evaluate alternative representations, graph types, retrieval methods, training approaches, or maintenance comparisons.

Abstract

from arXiv · show

Graph structures are increasingly used in dialog memory systems, but empirical findings on their effectiveness remain inconsistent, making it unclear which design choices truly matter. We present an experimental, system-oriented analysis of long-term dialog memory architectures. We introduce a unified framework that decomposes dialog memory systems into core components and supports both graph-based and non-graph approaches. Under this framework, we conduct controlled, stage-wise experiments on LongMemEval and HaluMem, comparing common design choices in memory representation, organization, maintenance, and retrieval. Our results show that many performance differences are driven by foundational system settings rather than specific architectural innovations. Based on these findings, we identify stable and reliable strong baselines for future dialog memory research. Code are available at https://github.com/AvatarMemory/UnifiedMem

1 Introduction

Prior studies disagree on whether graph-based dialog memory improves performance, while inconsistent system settings make those differences difficult to attribute. This paper therefore introduces a unified framework and controlled comparisons to identify reliable baselines.

  • Graph-based retrieval has been reported to improve dialog memory performance, but non-graph or lightweight approaches have achieved comparable or better results on similar benchmarks.
  • Different datasets, backbone models, evaluation settings, and underspecified implementation details limit reproducibility and complicate comparison.
  • Unaligned choices in memory representation, indexing, retrieval, and ranking make performance differences difficult to attribute to individual innovations.
  • The paper presents an experimental, system-oriented analysis rather than proposing a new method.
  • Its unified framework covers graph-based and non-graph approaches and supports controlled, stage-wise comparisons across memory design, organization, maintenance, indexing, and retrieval.
  • The analysis identifies stable strong baselines across both graph-based and non-graph systems for future research and development.

2 Related Work

Related work spans plain-index and graph-index dialog memory, as well as graph-structured retrieval in RAG. The paper connects these traditions while emphasizing that dialog memory additionally requires online maintenance of extracted memories.

  • RAG uses external knowledge retrieval to enrich prompts, while graph-based RAG adds entity relationships and multi-hop retrieval to the process.
  • Agent memory research includes plain-index and graph-index methods for sustaining long-term attention and acquiring knowledge from dialogue history.
  • Prior dialog memory work includes long-term-memory benchmarks, topic-driven summarization, and graph representations with entities as nodes and relations as edges.
  • RAG and agent memory increasingly share techniques, although RAG commonly indexes static raw content while dialog memory emphasizes extracted memories and online index maintenance.

3 A Unified Memory Framework

The unified framework represents dialog memory as components for keys, values, queries, indexing, and retrieval across four recurring pipeline stages. It accommodates flat and graph organizations, multiple maintenance operations, and distinct retrieval procedures.

  • The framework abstracts memory systems into a six-tuple containing keys, values, queries, an index structure, and a retrieval method.
  • The pipeline comprises memory extraction, memory indexing, memory retrieval, and question answering, which are repeatedly invoked as dialogue progresses.
  • Keys and Values: Keys are memory units such as summaries, facts, keywords, entities, triples, or raw sessions, while values are evidence supplied to the answering model.
  • Keys and Values: Raw sessions preserve contextual continuity but increase storage and reasoning costs; compressed derived information improves efficiency while risking information loss.
  • Flat Index and Key Organization: Separate organization stores each key as an independent vector, whereas merge organization combines keys by session or type to reduce fragmentation and redundancy.
  • Index Structure: Graph indices connect textual units or entities with edges representing semantic, structural, or temporal relationships, and may include hierarchical abstractions.
  • Index Maintenance and Update: Memory maintenance commonly uses Add, Update, Delete, and Noop operations; graph-based Update corresponds to node and edge merging.
  • Retrieval: Flat retrieval embeds the query, searches key vectors, maps selected keys to values, and re-ranks when keys and values are not one-to-one.

4 Experiments

The experiments compare dialog-memory design choices across LongMemEval and HaluMem using controlled, stage-wise evaluations. Results show that representation, maintenance, graph construction, activation, and model capacity each shape performance, with different settings favoring different outcomes.

  • Experimental setup: The study evaluates flat and graph-based memory systems on LongMemEval and HaluMem, covering representation, organization, maintenance, indexing, and retrieval choices.LongMemEval emphasizes retrieval and reasoning over long histories, while HaluMem evaluates extraction, updating, consistency, and hallucination avoidance; experiments use HaluMem-medium.
  • Memory representation and organization: Augmenting session keys with summaries, factual statements, and keywords consistently improves retrieval, with session,[S,F,K] recommended when raw session content is allowed.When raw content is disallowed, the authors recommend trying ([S,F,K]) first; separate keys perform best on HaluMem in additional results.
  • Memory maintenance: Update and Noop operations decrease memory precision but substantially improve recall and end-to-end QA accuracy.The authors attribute the precision decrease to potentially incorrect memory modifications and recommend considering both operations in practice.
  • Graph construction: Similarity-based graph expansion worsens retrieval because it introduces noise, whereas DescGraph performs better and entity descriptions improve entity–relation graphs.The proposed interpretation is that similarity edges need effective reranking to avoid noisy expansion.
  • Graph activation and ranking: Direct entity activation outperforms triple-based activation, while 1-hop expansion has marginal impact with augmented reranking but degrades retrieval when only Scores is used.For Value=session, the authors identify direct entity activation without expansion and ranking by (Scoree, Scoreg) as a strong baseline.
  • End-to-end comparison: On LongMemEval, graph methods outperform flat indexing for retrieval, but graph QA performance depends on value representation, model capacity, and dataset scale.Graphs perform better for Value=session, especially with higher-capacity construction models, but worse for Value=Key because flat keys aggregate richer session-derived information; weaker extraction models also reduce graph performance and recall.

5 Conclusion

The paper introduces a unified framework for controlled comparison of graph-based and non-graph dialog memory systems. Its results show that foundational settings substantially affect performance, while graph methods help only under suitable configurations.

  • The unified framework decomposes dialog memory systems into core components for controlled comparison across graph-based and non-graph approaches.
  • Foundational system settings substantially affect dialog memory performance.
  • Graph-based memory can help under certain configurations, but unsuitable graph construction or retrieval strategies may degrade results.
  • The framework and strong baselines clarify the design space of dialog memory systems.

Limitations

The framework covers most commonly used memory systems but not every customized architecture or design choice. Its conclusions are bounded by the two evaluated benchmarks and the limited backbone configurations tested.

  • The framework is intended to cover most commonly used memory systems, but customized architectures such as Memory OS fall outside its scope.
  • The experiments do not exhaustively evaluate alternative memory keys, heterogeneous or hierarchical graphs, advanced retrieval, additional training, or all maintenance operations.
  • Conclusions may vary on memory benchmarks with different characteristics, so configurations may reasonably differ across datasets and applications.
  • The evaluation covers only two backbone configurations and does not explore a broader range of model combinations.

Appendices

The appendices define the framework’s memory units, organization operations, graph elements, and expansion strategies. They distinguish fixed key-value mappings from genuine hierarchical graphs and describe common maintenance choices.

  • Memory representation: Keys may be summaries, statements, keywords, tags, entities, or triples used to represent dialog-derived information.
  • Memory index operations: Memory maintenance commonly uses Add, Update, Delete, and Noop operations.
  • Graph structure: Graph indices connect keys through explicit edges, including similarity, part-of, relational, temporal, and causal edges.
  • Graph structure: Graph nodes may represent chunks, sentences, entities, or hybrid structures, while hierarchical graphs connect multiple semantic levels.
  • Graph structure: The framework treats key-value relationships as fixed mappings rather than hierarchical edges, reserving hierarchical graphs for structures such as community-entity hierarchies.
  • Graph retrieval: Graph expansion can use simple neighbor traversal, structure-aware ranking, or semantic-aware expansion followed by query-based rescoring.

B Dataset Details

The evaluation uses LongMemEval and HaluMem to assess complementary aspects of long-term dialog memory. Method comparisons primarily rely on LongMemEval because it provides ground-truth retrieval metrics and avoids the exclusive reliance on LLM judging in HaluMem.

  • LongMemEval: LongMemEval evaluates retrieval and reasoning over extremely long dialogs across extraction, cross-session, temporal, updating, and refusal tasks.
  • HaluMem: HaluMem evaluates memory extraction, updating, consistency, and resistance to hallucinated responses under many information updates.
  • Evaluation protocol: HaluMem metrics rely on LLM-as-Judge, whereas LongMemEval supplies relevant-session ground truth and retrieval metrics such as recall and NDCG.
  • Appendix analysis: The appendix extends the end-to-end results and reports ancillary analyses of trade-offs behind minor design configurations.

C.1 Full Evaluation Metrics

The evaluation compares flat and graph indexing across two model environments and six LongMemEval question dimensions. Graph-based retrieval can outperform on retrieval metrics, yet its QA benefit varies with task and system setting, while incomplete reporting limits comparisons with HaluMem results.

  • Evaluation setup: The evaluation spans four configurations across six dimensions: Single Session User, Single Session Preference, Knowledge Update, Single Session Assistant, Temporal Reasoning, and Multi-Session.The configurations vary indexing methodology and LLM scale/capacity.
  • Evaluation setup: Setting 1 uses Llama-3.1-8B with Contriever, whereas Setting 2 uses GPT-4o-mini with text-embedding-3-small.Setting 1 represents the more constrained local environment; GPT-4o evaluates all experiments.
  • Evaluation findings: Graph-setting2 shows an unexpected performance drop on Single Session Assistant questions because the graph is built from user messages while answers lie in assistant responses.The information-source mismatch leaves the graph without necessary answer content.
  • Evaluation findings: The supplementary HaluMem results report that the proposed strong baselines consistently outperform Table C.2 results, particularly on Memory Recall.The unspecified extraction and embedding settings of prior HaluMem methods make the differences difficult to attribute.

C.2 Cost-Efficiency Analysis

The analysis finds that flat memory is cheaper to construct and retrieve, while graph memory adds scale-dependent but manageable overhead. It also evaluates key organization, update operations, and source filtering as practical system choices.

  • Cost-efficiency: Graph retrieval is slower than flat retrieval, with 574 ms versus 240 ms per query on LongMemEval-M, but remains practically acceptable at over 50k sessions.On LongMemEval-S, latency is comparable at 45 versus 44 ms per query.
  • Cost-efficiency: Graph extraction takes about 2.1 seconds per LongMemEval-M session versus 0.5 seconds for flat memory, while extraction is usually asynchronous.The added graph cost comes from entity extraction and description aggregation.
  • Cost-efficiency: On LongMemEval-M, both memory types compress over 100M dialog tokens into fewer than 30M stored-memory tokens, although graph memory stores somewhat more.The comparison is based on unique sessions and cached extraction.
  • Design choices: The separate key strategy outperforms merge strategies on HaluMem retrieval recall, especially when raw session content is unavailable.The reported results favor maintaining independent keys under constrained retrieval.
  • Design choices: Including add with update and noop substantially improves memory-update accuracy and end-to-end QA while leaving extraction metrics relatively stable.The full operation set is adopted for subsequent update experiments.
  • Design choices: Prejudge filters irrelevant or redundant chunks before fine-grained extraction and graph updates, slightly improving retrieval metrics while reducing downstream computational overhead.The mechanism is enabled in the paper’s experimental configurations.

D.2 Comparison of System Settings

The unified comparison shows that inconsistent foundational settings complicate attribution across dialog-memory systems. Within the paper’s framework, systemic refinement yields strong results, while graph retrieval’s advantage does not consistently translate into QA accuracy.

  • System comparison: Existing studies vary in extraction models, embedding models, top-k constraints, and stored value types, making architecture-level comparisons difficult.These system-level discrepancies prevent clean isolation of memory or retrieval paradigms.
  • System comparison: The framework reports 0.969 retrieval recall on LME-S and end-to-end QA accuracy of 0.892 and 0.754 on the S and M subsets using gpt-4o-mini.The paper presents these results as surpassing contemporary baselines.
  • Graph versus flat: Graph-based indexing achieves superior retrieval performance, but in the V = Session setting it matches the flat baseline and in V = Key it underperforms on downstream QA.The paper investigates representative cases to explain this retrieval-to-QA discrepancy.
  • Graph versus flat: For capacity-limited models, graph-retrieved context can hurt QA when temporal metadata distracts reasoning or atomic entities fragment semantic context.These failure mechanisms are illustrated in the V = Session and V = Key cases.
  • Graph versus flat: Graph indexing can still help when it reduces interference or captures long-range dependencies across disjoint sessions.The success cases include knowledge update and multi-session queries.

E.3 Case Study: Entity Name vs. Entity Description as Keys

The key-comparison case study finds that entity descriptions provide more informative retrieval keys than entity names for temporal and event-specific recall. Graph and flat indexes each succeed in different conditions, depending on contextual coherence and cross-session structure.

  • Entity keys: Entity-name keys tend to retrieve broad concepts and abstract durations rather than grounding temporal entities in specific events.Examples include generic recipe categories and an ungrounded “couple of days” duration.
  • Entity keys: Entity-description keys link “Last Weekend” to the event of baking a chocolate cake, bridging the query’s semantic gap to the stored memory.The description preserves the specific event needed for retrieval.
  • Index behavior: A flat index can outperform a graph index when graph retrieval adds distracting temporal information or fragments relevant entities into semantically sparse units.The flat index succeeds by preserving richer notes and semantic cohesion in the illustrated failures.
  • Entity keys: For complex long-term memory tasks involving temporal reasoning and specific event recall, descriptions capture more useful who, what, and when information than names.The case study concludes that names can fall back on redundant or overly categorized labels.
  • Index behavior: A graph index can outperform flat retrieval when the query requires aggregating evidence across disjoint sessions or when flat-context noise interferes with answering.The reported success cases involve multi-session dependencies and lower retrieval interference.
Loading 2601.01280v3…