Source-linked AI summary
HyperMem: Hypergraph Memory for Long-Term Conversations
Juwei Yue, Chuanrui Hu, Jiawei Sheng, Zuyi Zhou, Wenyuan Zhang, Tingwen Liu, Li Guo, Yafeng Deng
TL;DR
Long-term conversational memory must preserve joint associations that pairwise RAG and graph methods fragment. HyperMem uses a hierarchical hypergraph of topics, episodes, and facts with lexical-semantic indexing and coarse-to-fine retrieval, achieving 92.73% LLM-as-a-judge accuracy on LoCoMo. The design is evaluated in a single-user setting, while multi-user access control and memory isolation remain open challenges.
Problem
Existing RAG and graph-based memory methods rely mainly on pairwise relations, limiting their ability to capture high-order associations in long-term conversations.
Method
HyperMem organizes topics, episodes, and facts with hyperedges, then applies lexical-semantic indexing and coarse-to-fine retrieval.
Results
92.73% LLM-as-a-judge accuracy on LoCoMo gives HyperMem state-of-the-art performance for long-term conversations.
Takeaways & Limitations
HyperMem unifies scattered dialogue content into coherent units for efficient retrieval of high-order associations.
Takeaways & Limitations
The current design assumes a single-user scenario, leaving access control and memory isolation challenging for multi-user or multi-agent settings.
Abstract
from arXiv · showhide
Long-term memory is essential for conversational agents to maintain coherence, track persistent tasks, and provide personalized interactions across extended dialogues. However, existing approaches as Retrieval-Augmented Generation (RAG) and graph-based memory mostly rely on pairwise relations, which can hardly capture high-order associations, i.e., joint dependencies among multiple elements, causing fragmented retrieval. To this end, we propose HyperMem, a hypergraph-based hierarchical memory architecture that explicitly models such associations using hyperedges. Particularly, HyperMem structures memory into three levels: topics, episodes, and facts, and groups related episodes and their facts via hyperedges, unifying scattered content into coherent units. Leveraging this structure, we design a hybrid lexical-semantic index and a coarse-to-fine retrieval strategy, supporting accurate and efficient retrieval of high-order associations. Experiments on the LoCoMo benchmark show that HyperMem achieves state-of-the-art performance with 92.73% LLM-as-a-judge accuracy, demonstrating the effectiveness of HyperMem for long-term conversations.
1 Introduction
HyperMem addresses fragmented long-term conversational retrieval by modeling high-order associations with a three-level hypergraph and hyperedges. It combines lexical-semantic indexing with coarse-to-fine retrieval and achieves 92.73% LLM-as-a-judge accuracy on LoCoMo.
- Existing RAG and graph-based memory methods rely on pairwise relations, limiting their ability to capture joint dependencies among multiple conversational elements.
- HyperMem models topics, episodes, and facts with hyperedges that group related content into coherent units.
- The architecture combines lexical cues, dense semantic propagation, and coarse-to-fine retrieval from topics to episodes to facts.
- 92.73% LLM-as-a-judge accuracy on LoCoMo establishes HyperMem’s state-of-the-art performance for long-term conversations.
2 Related works
Related work spans retrieval-based, graph-based, hierarchical, agentic, and latent-memory systems. HyperMem differs by applying hyperedges and topic-guided hierarchical retrieval to continuously evolving conversational memory.
- GraphRAG and related methods use graph topology for structure-aware reasoning and multihop retrieval, while hierarchical systems organize or compress retrieved information.
- Prior hypergraph methods target static knowledge bases and generally lack hierarchical retrieval that preserves coherence across extended dialogues.
- Existing long-term memory agents use structured, graph-based, hierarchical, compressed, or learned representations to preserve and retrieve information across sessions.
- HyperMem explicitly groups topically related memories with hyperedges and uses topic-guided hierarchical retrieval across temporal gaps.
3 Approach
HyperMem represents long-term conversational memory as a three-level hypergraph linking topics, episodes, and facts. This structure supports high-order relational modeling and precise retrieval targets.
- HyperMem uses hyperedges to connect multiple nodes, modeling higher-order associations beyond conventional pairwise graphs.
- The memory hierarchy contains topic nodes for themes, episode nodes for coherent temporal segments, and fact nodes for atomic retrieval targets.
- Topic-level hyperedges connect episodes sharing a topic, while episode-level hyperedges connect facts belonging to the same episode.
3.2 Hypergraph Memory Construction
HyperMem constructs memory incrementally by detecting coherent episodes, aggregating temporally scattered episodes into topics, and extracting query-oriented facts. Hyperedges preserve narrative connections while lexical-semantic indexing supports targeted retrieval.
- The construction pipeline detects episode boundaries, aggregates related episodes into topics, and extracts informative facts grounded in episode context.
- 3.2.1 Episode Detection: Episode detection uses semantic completeness, time gaps, and linguistic transition signals to segment dialogue into coherent memory units.
- 3.2.1 Episode Detection: Streaming episode segmentation reduces irrelevant context and improves topic organization and retrieval convenience.
- 3.2.2 Topic Aggregation: Topic aggregation links episodes that share a theme, allowing narratives spanning weeks or months to be retrieved despite temporal fragmentation.
- 3.2.3 Fact Extraction: Facts provide compact, query-oriented assertions with potential query patterns and keywords, while retaining links to their source episodes.
3.3 Hypergraph Memory Retrieval
HyperMem retrieves long-term conversational memories by combining lexical-semantic indexing with hypergraph-based propagation and a hierarchical topic-to-episode-to-fact search. This design aligns related memories and preserves query-relevant context while reducing response-context size.
- Online Retrieval Strategy: HyperMem traverses from topics to episodes to facts, progressively selecting relevant candidates for structured retrieval.The process uses topic, episode, and fact stages with top-k selection at each level.
- Online Retrieval Strategy: Dual BM25 and dense semantic indices combine lexical cues with semantic intent for topic, episode, and fact retrieval.Rankings are fused with Reciprocal Rank Fusion and refined by a reranker.
- Offline Index Construction: Hypergraph propagation enriches node embeddings by aggregating information from incident hyperedges, aligning semantically related memories.The propagation strength is controlled by λ, and the mechanism supports retrieval of high-order associations.
- Offline Index Construction: Hyperedges connect related episodes and facts into shared topical structures, preserving associations that pairwise representations may fragment.Importance weights can be assigned during aggregation based on narrative contribution.
- Final Response Generation: Retrieved fact content, optionally augmented with episode summaries, forms a compact response context that preserves answerable information while reducing token consumption.The constructed context is passed to the conversational agent for final response generation.
4 Experiments
On LoCoMo, HyperMem achieves the strongest overall accuracy among compared RAG and memory systems, while its hierarchical design supports category-level gains and efficient retrieval. Ablations and sensitivity analyses identify episode context, topic coverage, and balanced retrieval components as important contributors.
- 4.2 Main Results: 92.73% overall accuracy makes HyperMem the best method, exceeding HyperGraphRAG by 6.24% and MIRIX by 7.35%.Table 1 reports LLM-as-a-judge accuracy on LoCoMo.
- 4.2 Main Results: HyperMem reaches 96.08% on Single-hop, 93.62% on Multi-hop, and 89.72% on Temporal questions, while Open Domain remains challenging.The reported gains are attributed to structured fact retrieval, hyperedge-based episode binding, and temporal anchors.
- 4.2 Main Results: Hyperedges bind related episodes and topic-guided retrieval narrows candidates, filtering irrelevant context while preserving temporal coherence.These two designs are identified as the source of the reported improvements.
- 4.3 Ablation Study: Removing Episode context causes the largest ablation loss, reducing overall performance by 3.76% and Temporal performance by 5.61%.Flattening retrieval to facts alone reduces Multi-hop performance by 5.68%, while removing Fact context reduces it by 2.84%.
- 4.4 Hyperparameter Analysis: A fusion weight α = 0.5 achieves 92.66%, while increasing topic top-k from 1 to 10 raises accuracy from 76.88% to 92.66%.Episode top-k is comparatively stable, and fact top-k peaks at k=30 before higher values introduce slight degradation.
- 4.5 Efficient Analysis: HyperMem reaches 92.73% accuracy at 7.5× tokens, whereas GraphRAG reaches 67.60% at 35.3× and HyperGraphRAG reaches 86.49% at 26.3×.Fact Only reaches 89.48% at 2.5× tokens, and Episode + Fact outperforms it by 3–4%.
5 Conclusion
HyperMem models high-order associations in long-term conversational memory through a hierarchical hypergraph and achieves 92.73% LLM-as-a-judge accuracy on LoCoMo. The source code is reported as forthcoming.
- 92.73% LLM-as-a-judge accuracy on LoCoMo establishes HyperMem’s reported state-of-the-art result for long-term conversations.The conclusion attributes this performance to HyperMem’s hypergraph-based agentic memory architecture.
- HyperMem organizes topics, episodes, and facts with hyperedges to unify scattered dialogue content into coherent units.Its indexing combines lexical cues with hypergraph embedding propagation, followed by coarse-to-fine retrieval.
- The source code is reported as about to be released.
Limitations
HyperMem is currently designed for single-user scenarios, while broader deployments raise access-control and memory-isolation challenges. Open Domain questions also remain difficult when they require knowledge beyond conversation history.
- HyperMem assumes a single-user setting, creating access-control and memory-isolation challenges for multi-user or multi-agent extensions.
- Open Domain questions remain challenging when answering them requires external knowledge beyond the conversation history.
A Algorithm
The appendix provides detailed pseudocode for HyperMem’s core procedures in Algorithms 1, 2, and 3.
- Algorithms 1, 2, and 3 provide detailed pseudocode for HyperMem’s core procedures.
B Prompt Templates
The prompt-template appendix documents prompts for episode boundary detection, topic aggregation, and fact extraction. These templates correspond to the main stages of HyperMem’s memory construction process.
- The prompt templates cover episode boundary detection, topic aggregation, and fact extraction.Figures 6, 7, and 8 describe the prompts for these three operations, respectively.
C Case Study
HyperMem addresses diverse LoCoMo query types by organizing related episodes and facts under hierarchical topic structures. Across single-hop, multi-hop, temporal, and open-domain cases, it retrieves or synthesizes answers more accurately than the cited baselines.
- Single-Hop Task: HyperMem retrieves the exact fact that Maria started volunteering at a local dog shelter, while GraphRAG confuses it with “homeless shelter” and retrieves “aerial yoga” from another period.Hierarchical retrieval navigates Topic and Episode layers to locate the matching Fact.
- Multi-Hop Task: Seven tournaments across 10 months are correctly aggregated, whereas GraphRAG identifies only “at least two” because pairwise edges fragment related memories.HyperMem’s Topic hyperedge groups all tournament-related episodes and supports precise dates for each.
- Temporal Reasoning Task: One pet dog named Toby is correctly identified at the queried time, while GraphRAG confuses people and HyperGraphRAG overcounts with four pets.The Episode layer preserves temporal anchors for state reconstruction.
- Open Domain Task: HyperMem correctly infers that John would not move abroad by synthesizing evidence about his U.S.-anchored military and political goals.HyperGraphRAG instead answers yes from superficial travel mentions.
- Memory Construction and Retrieval: The memory-construction pipeline detects episode boundaries, aggregates related topics, extracts facts, and links facts back to their source episodes.Offline indexing builds BM25 and vector indexes, while online retrieval proceeds from topics to episodes.