PaperlayerView exact revision on arXiv

Source-linked AI summary

PersonalAI: A Systematic Comparison of Knowledge Graph Storage and Retrieval Approaches for Personalized LLM agents

Mikhail Menschikov, Dmitry Evseev, Victoria Dochkina, Ruslan Kostoev, Ilia Perepechkin, Petr Anokhin, Nikita Semenov, Evgeny Burnaev

arXiv:2506.17001v6cs.CLcs.IR

TL;DR

Personalized LLM agents need structured long-term memory because RAG has limited support for semantic relationships across stored interactions. The paper builds an automatically constructed knowledge graph with configurable storage and retrieval, then evaluates it across three benchmarks and multiple model configurations. Results show that retrieval strategies and traversal constraints produce task- and model-dependent performance, with BeamSearch and hybrid methods prominent among strong configurations.

  • Problem

    RAG improves factual recall but remains unstructured and weakly supports semantic relationships across long-term stored memories.

  • Method

    The paper extends AriGraph with a flexible knowledge-graph memory containing object, thesis, and episodic representations plus configurable retrieval algorithms.

  • Results

    Performance varies systematically by retrieval strategy and graph traversal constraints across DiaASQ, HotpotQA, and TriviaQA; BeamSearch and hybrid methods frequently appear in strong configurations.

  • Takeaways & Limitations

    The framework provides an extensible basis for adapting structured memory and retrieval to different tasks and LLM scales.

Abstract

from arXiv · show

Personalizing language models by effectively incorporating user interaction history remains a central challenge in the development of adaptive AI systems. While large language models (LLMs), combined with Retrieval-Augmented Generation (RAG), have improved factual accuracy, they often lack structured memory and fail to scale in complex, long-term interactions. To address this, we propose a flexible external memory framework based on a knowledge graph that is constructed and updated automatically by the LLM. Building upon the AriGraph architecture, we introduce a novel hybrid graph design that supports both standard edges and two types of hyper-edges, enabling rich and dynamic semantic and temporal representations. Our framework also supports diverse retrieval mechanisms, including A*, WaterCircles traversal, beam search, and hybrid methods, making it adaptable to different datasets and LLM capacities. We evaluate our system on TriviaQA, HotpotQA, DiaASQ benchmarks and demonstrate that different memory and retrieval configurations yield optimal performance depending on the task. Additionally, we extend the DiaASQ benchmark with temporal annotations and internally contradictory statements, showing that our system remains robust and effective in managing temporal dependencies and context-aware reasoning

I. INTRODUCTION

The paper addresses the challenge of storing and retrieving structured long-term interaction history for personalized LLM agents. It proposes a flexible knowledge-graph memory with configurable retrieval and evaluates how configurations perform across tasks and model scales.

  • RAG improves factual recall but remains unstructured and weakly represents semantic relationships across stored memories.
  • The framework organizes nodes, knowledge triples, thesis statements, and episodic traces into a customizable knowledge graph for long-term memory.
  • Multiple retrieval mechanisms, including A*, WaterCircles, BeamSearch, and hybrid strategies, adapt retrieval to task demands and model capacity.
  • The system extends AriGraph by allowing independent customization of memory construction and retrieval modules for task-specific evaluation.
  • The paper evaluates six retrieval methods, reports performance across datasets, and augments DiaASQ with temporal structures for temporal reasoning.

III. METHODS

The memory graph combines semantic and episodic representations to encode concepts, relationships, complete thoughts, and source passages. Standard edges and hyper-edges connect these elements into a structured external memory.

  • The memory model separates semantic memory from episodic memory and represents both with vertices and edges.Semantic memory contains object and thesis components, while episodic memory contains passage-linked components.
  • Object vertices represent atomic concepts extracted from source texts, and object edges encode direct text-attributed relationships as triples.
  • Thesis vertices encapsulate complete atomic thoughts expressed in source texts.
  • Thesis edges act as hyper-edges linking object vertices extracted from the same source text and associated with a thesis.
  • Episodic vertices correspond to original text passages and serve as hyper-edges linking related vertices.

B. MEMORY CONSTRUCTION

Memory construction converts weakly structured text into a graph through vertex and edge formation, LLM-generated hyper-edges, and output parsing. The QA pipeline then matches question entities, traverses the graph, filters triples, and generates an answer.

  • B. MEMORY CONSTRUCTION: The Memorize pipeline constructs memory in three steps: forming vertices and edges, generating LLM-based hyper-edges, and parsing structured outputs.
  • B. MEMORY CONSTRUCTION: The graph fragment represents object, thesis, and episodic vertices extracted from natural-language text.
  • B. MEMORY CONSTRUCTION: Newly extracted triples are matched against existing vertices, followed by breadth-first traversal of associated standard and hyper-edges to identify outdated information.
  • C. INFORMATION SEARCH IN MEMORY: The QA pipeline extracts question entities, matches them to graph vertices, retrieves candidate triples, filters them by semantic similarity, and generates an answer.
  • C. INFORMATION SEARCH IN MEMORY: Entity-to-vertex matching targets a relevant subgraph because the information needed for answering is typically localized around question entities.

D. RETRIEVAL ALGORITHMS

The retrieval system balances relevance and completeness through configurable graph-traversal algorithms and parameters. It combines shortest-path, beam, WaterCircles, and mixed strategies to retrieve answer-supporting triples.

  • Retrieval algorithms balance relevance and completeness by dynamically traversing the memory graph with configurable parameters.
  • A*: A* extracts deduplicated triples along shortest paths using inner-product and weighted shortest-path heuristics.
  • WaterCircles: Modified BFS initializes from question-matched vertices, counts entities in text fragments, and ranks thesis and episodic triples by entity intersections.
  • BeamSearch: BeamSearch generates multiple semantically relevant paths under limits on depth, path count, vertex revisitation, and path overlap.
  • Mixed Algorithm: The Mixed Algorithm unions outputs from A*, WaterCircles, and BeamSearch so information missed by one method may be retrieved by another.

IV. EXPERIMENT SET-UP

The evaluation uses DiaASQ, HotpotQA, and TriviaQA to test the framework across temporal, multi-hop, factoid, and general-domain question-answering challenges.

  • The framework is evaluated on DiaASQ, HotpotQA, and TriviaQA to cover varied domains, structural complexities, and reasoning requirements.
  • DiaASQ contains mobile-device forum dialogues with structured true statements used to generate evaluation questions.
  • DiaASQ is extended with temporal annotations and internally contradictory statements to test temporal dynamics and contradictory information.
  • HotpotQA tests multi-hop reasoning across documents, while TriviaQA tests factoid-style open-domain knowledge retrieval.
  • Manageable dataset subsets were created by filtering contexts and segmenting long TriviaQA documents for practical iterative experimentation.
  • The prepared benchmarks support comparison of PersonalAI’s memorization and question-answering functionality with RAG and GraphRAG baselines.

B. MODELS

The experiments vary language-model scale, traversal algorithms, and graph-vertex restrictions to study how retrieval configurations affect the QA pipeline.

  • The study evaluates 7B/8B and larger-than-14B language models for memory construction and question answering.
  • The evaluated models include Qwen2.5 7B, DeepSeek R1 7B, Llama3.1 8B, GPT-4o-mini, and DeepSeek V3.
  • Traversal methods include A*, WaterCircles, BeamSearch, and the hybrids WC + BS, A* + BS, and A* + WC.
  • Traversal constraints can exclude episodic, thesis, or object vertices, while “all” leaves graph traversal unrestricted.

E. SUMMARY OF EXPERIMENT CONFIGURATIONS

Across 308 QA configurations, performance and efficiency depend on model scale, retrieval strategy, graph constraints, and vector-database choice.

  • Evaluation design: 308 QA configurations were evaluated across DiaASQ, HotpotQA, and TriviaQA, using 100 question-answer pairs per dataset/configuration.
  • System configuration: The memory system combines graph and vector components, with Milvus, Redis, and MongoDB used during experiments.
  • System configuration: The outdated-knowledge deletion stage was disabled, and text ingestion averaged approximately 1.35 fragments per minute for 550–650-character inputs.
  • Evaluation design: The LLM-as-a-Judge framework was adopted because conventional metrics and BERTScore did not sufficiently distinguish semantically different answers.
  • Accuracy results: BeamSearch frequently supports strong configurations, while larger DeepSeek V3 configurations often combine BeamSearch with WaterCircles.
  • Traversal constraints: For 7B/8B models, approximately 74% of lowest-quality configurations restricted thesis vertices, whereas high-quality configurations often restricted episodic or object vertices.
  • Robustness: A* plus WaterCircles degraded by at most 4% for 8B models, while BeamSearch scores varied by as much as 24% under suboptimal constraints.

VII. CONCLUSION

The framework extends AriGraph with structured memory and multiple retrieval strategies, whose effectiveness varies with model scale, task, and traversal constraints.

  • PersonalAI extends AriGraph with object, episodic, and thesis vertices plus hyper-edges for richer temporal and semantic memory representations.
  • The system supports A*, WaterCircles, BeamSearch, and hybrid retrieval methods adaptable to model scale and task requirements.
  • Across DiaASQ, HotpotQA, and TriviaQA, performance varies systematically with retrieval strategy and graph traversal constraints.
  • For 7B–8B models, high-accuracy configurations restrict episodic or object vertices and rely on BeamSearch, whereas larger models benefit from BeamSearch–WaterCircles hybrids.
  • Thesis vertices often encode critical information, and excluding them typically degrades performance, especially for 7B models.

VIII. FUTURE WORK

Future work targets finer temporal control, lower retrieval latency, and more robust, private, distributed memory infrastructure for personalized agents.

  • Temporal dynamics: A proposed “memory time” parameter would filter triples by temporal proximity and edge type.The goal is to prioritize temporally proximate data or selected relationship categories.
  • Retrieval efficiency: Future traversal improvements target vector-storage bottlenecks through approximate nearest neighbor search while maintaining comparable precision.
  • Retrieval efficiency: More selective aggregation into smaller vector stores is intended to reduce search space and accelerate vector retrieval.
  • Distributed storage: Erasure-coded and locally recoverable layouts are proposed for sharding graph and vector indices across nodes.These layouts aim to support fast repair and continued operation during partial server unavailability.
  • Privacy and verification: The framework also plans private and verifiable retrieval protocols so agents can query remote memory without revealing user intent.Result verification is intended to detect incorrect or malicious responses.

APPENDIX A LLM PROMPTS USED TO BUILD MEMORY GRAPH BY MEMORIZE PIPELINE

The appendices document prompts for constructing, updating, and querying the memory graph, along with preprocessing steps used to form evaluation subsets.

  • APPENDIX A LLM PROMPTS USED TO BUILD MEMORY GRAPH BY MEMORIZE PIPELINE: Tables 7 and 8 provide Memorize prompts for extracting thesis and object memories as triples from natural-language text.These prompts transform unstructured text into structured graph representations.
  • APPENDIX B LLM PROMPTS USED TO FIND OUTDATED INFORMATION IN CONSTRUCTED MEMORY GRAPH: Tables 9 and 10 provide prompts for identifying stale thesis-related and object memories in the constructed graph.
  • APPENDIX C LLM PROMPTS USED IN PROPOSED QA PIPELINE: Tables 11 and 12 document QA prompts for extracting key entities and generating contextually appropriate answers.
  • APPENDIX D PREPROCESSING OPERATIONS FOR EVALUATION DATASETS: HotpotQA preprocessing selected the distractor/validation subset, filtered contexts to 64–1024 characters, and retained 13,291 contexts.
  • APPENDIX D PREPROCESSING OPERATIONS FOR EVALUATION DATASETS: TriviaQA preprocessing segmented long contexts with RecursiveCharacterTextSplitter using 1024-character chunks and 64-character overlap.
  • APPENDIX D PREPROCESSING OPERATIONS FOR EVALUATION DATASETS: The resulting HotpotQA, TriviaQA, and DiaASQ subsets formed the evaluation sets summarized in Table 13.

APPENDIX E RETRIEVAL HYPERPARAMETERS

The retrieval appendix specifies concrete settings for A*, WaterCircles, and BeamSearch, alongside deterministic LLM inference for reproducible evaluation.

  • A*: A* uses inner-product similarity, a maximum depth of 10, and a maximum of 150 passed nodes.
  • WaterCircles: WaterCircles uses strict filtering with 15 hyper-triples, 15 episodic triples, 25 chained triples, and 6 other triples.
  • BeamSearch: BeamSearch uses depth 5, up to 10 paths, disabled path-intersection restrictions, mean alpha 0.75, and mixed final sorting.
  • Evaluation inference: LLM evaluation uses deterministic generation with 2048 predictions, seed 42, temperature 0.0, and top-k 1.Qwen2.5 7B evaluates whether pipeline answers are correct.
  • Evaluation inference: Table 14 contains the prompts used in the LLM-as-a-Judge assessment.

APPENDIX G CHARACTERISTICS OF CONSTRUCTED MEMORY GRAPHS

The appendix characterizes graph construction, parsing quality, storage, and QA configurations, then compares selected configurations with RAG and GraphRAG baselines.

  • Graph construction: The QA evaluation constructed 14 memory graphs, with their structural characteristics reported in Table 15.
  • Parsing and graph composition: Average parsing errors were 7.0% for DiaASQ, 6.3% for HotpotQA, and 7.3% for TriviaQA, causing incomplete contextual storage in some graphs.
  • Parsing and graph composition: DeepSeek R1 7B and Qwen2.5 7B produced the most comprehensive graph representations, while Qwen2.5 7B achieved the finest contextual granularity.
  • Construction cost: Memorize pipeline timing and speed are reported in Table 18, with GPT-4o-mini configurations showing the highest parsing and saving speed at 1.86.
  • Construction cost: A memory graph occupies approximately 80–90 GB with Milvus but approximately 4–6 GB with Qdrant on the selected datasets.
  • QA configurations: Qwen2.5 reached 0.2 among 7B models, while DeepSeek V3 setups reached the highest overall effectiveness at 0.56.Top 7B configurations predominantly used A* with episodic traversal restricted; leading DeepSeek V3 setups often combined BeamSearch and WaterCircles.
  • Baseline comparison: The proposed method improved 14.1% over GraphRAG in one configuration but underperformed standard RAG by 17.8% against in-domain fine-tuned baselines.The comparison used configurations described for TriviaQA and HotpotQA in Table 20.
Loading 2506.17001v6…