PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents

Haoran Sun, Shaoning Zeng

arXiv:2507.22925v1cs.CLcs.AI

TL;DR

Existing LLM-agent memory methods do not fully provide structured organization and efficient retrieval for long-term reasoning. H-MEM uses semantic-abstraction layers with positional indices and hierarchical routing, and it consistently outperforms five baselines across five LoCoMo task settings while achieving high retrieval efficiency. Its current scope is primarily text-based, leaving multimodal memory insufficiently supported.

  • Problem

    Existing memory mechanisms have shortcomings in structured organization and efficient systematic retrieval for long-term LLM-agent reasoning.

  • Method

    H-MEM organizes and updates memory across four semantic-abstraction layers, using positional indices to route retrieval toward related sub-memories.

  • Results

    H-MEM consistently achieves the highest average F1 and BLEU-1 scores across model and task configurations, improving over baselines by 14.98 and 12.77 points, respectively.

  • Takeaways & Limitations

    H-MEM provides structured long-term memory access with reported advantages in preserving, accessing, and efficiently retrieving conversational memory.

  • Takeaways & Limitations

    The current architecture primarily supports text-based memory and has limited support for images, audio, and video in multimodal dialogue.

Abstract

from arXiv · show

Long-term memory is one of the key factors influencing the reasoning capabilities of Large Language Model Agents (LLM Agents). Incorporating a memory mechanism that effectively integrates past interactions can significantly enhance decision-making and contextual coherence of LLM Agents. While recent works have made progress in memory storage and retrieval, such as encoding memory into dense vectors for similarity-based search or organizing knowledge in the form of graph, these approaches often fall short in structured memory organization and efficient retrieval. To address these limitations, we propose a Hierarchical Memory (H-MEM) architecture for LLM Agents that organizes and updates memory in a multi-level fashion based on the degree of semantic abstraction. Each memory vector is embedded with a positional index encoding pointing to its semantically related sub-memories in the next layer. During the reasoning phase, an index-based routing mechanism enables efficient, layer-by-layer retrieval without performing exhaustive similarity computations. We evaluate our method on five task settings from the LoCoMo dataset. Experimental results show that our approach consistently outperforms five baseline methods, demonstrating its effectiveness in long-term dialogue scenarios.

1 Introduction

LLM Agents need memory mechanisms to integrate historical interactions in long-term dialogue, but existing approaches face limitations in structured organization and efficient retrieval. H-MEM addresses these limitations with hierarchical memory, positional indexing, and a memory-update mechanism, and is evaluated against five baselines on five LoCoMo tasks.

  • Long-term LLM-agent reasoning depends on integrating historical interactions through effective memory mechanisms.
  • Concatenating all previous interactions is constrained by the LLM context window, while layered and networked memories introduce retrieval, consistency, or accuracy trade-offs.
  • H-MEM organizes memory into semantic-abstraction levels and embeds positional indices linking each level to related subordinate memories.
  • The architecture is designed to reduce irrelevant information during retrieval, lowering computational costs while enabling targeted memory access.
  • H-MEM is compared with five baseline methods across five LoCoMo question-answering task categories and multiple language models.

2 Related Work

Prior work extends LLM context and memory capabilities through layered storage, interactive retrieval, memory controllers, structured attributes, and evolving knowledge networks. However, existing methods still leave gaps in systematic memory organization and efficient retrieval, motivating further development.

  • ReadAgent, MemGPT, and SCM extend long-term processing through page-level compression, virtual context management, or dynamic memory control.
  • MemInsight improves memory representation by extracting key information and generating attributes for semantic retrieval across several tasks.
  • MemoryBank combines memory storage, retrieval, and updating with a forgetting-curve-inspired mechanism for long-term interaction.
  • Despite these advances, MemoryBank is described as limited in structured memory storage and efficient systematic retrieval.

3 Method

H-MEM organizes memory into a four-level hierarchy whose abstract layers index increasingly fine-grained episodic content. Its positional pointers support top-down retrieval, adaptive updates, and lower-complexity search than flat exhaustive retrieval.

  • Hierarchical Storage: H-MEM stores memories across Domain, Category, Memory Trace, and Episode layers, progressing from abstract indices to complete episodic content and user profiles.The first three layers provide progressively refined organization, while the Episode Layer stores contextual interaction memory.
  • Index Encoding: A memory entry combines a dense semantic vector, its self-position index, and pointers to semantically related sub-memories in the next layer.These pointers enable index-based routing without exhaustive similarity computation.
  • Hierarchical Retrieval: During inference, H-MEM embeds the query, selects top-k memories at the highest abstraction layer, and recursively follows associated pointers toward fine-grained memories.Similarity is calculated only between the query and semantic memory vectors, while positional encodings guide subsequent traversal.
  • Adaptive Hierarchy: The system uses a four-layer configuration selected to balance retrieval accuracy and efficiency, while allowing users to add or remove levels based on conversational complexity.Simpler conversations may use fewer levels, whereas complex multi-topic conversations may use more.
  • Computational Complexity: Under the stated fixed conditions, flat retrieval costs O(a · 10^6 · D), whereas H-MEM retrieval is approximated by O((a + k · 300) · D).The comparison assumes a domains, 100 categories per domain, 100 traces per category, and 100 episodes per trace.
  • Memory Updating: H-MEM adjusts memory weights using user feedback: approval strengthens a memory, no feedback follows natural shrinkage, and rebuttal reduces its weight.The update mechanism is intended to reflect changing psychological states and possible memory expiration.

4 Experiment

H-MEM is evaluated on five LoCoMo QA task categories across multiple models, baselines, and memory-retrieval efficiency settings. It consistently improves answer quality while reducing retrieval latency and computation, with ablations supporting the joint importance of hierarchical storage and retrieval.

  • Experimental Setup: Experiments use the LoCoMo dataset, multiple foundation models, five long-term dialogue QA tasks, and five comparison methods.The evaluation reports F1 and BLEU-1, with H-MEM compared against LoCoMo, ReadAgent, MemoryBank, MemGPT, and A-MEM.
  • Comparison to Baselines: H-MEM consistently achieves the highest average F1 and BLEU-1 scores across model and task configurations, improving over baselines by 14.98 and 12.77 points, respectively.The method also maintains leading performance across 1.5B, 3B, and 7B model scales.
  • Computational Efficiency: At maximum memory load, H-MEM inference remains below 100 ms, while the baseline exceeds 400 ms, making the baseline 5 times slower.The comparison evaluates retrieval cost, latency, answer quality, and accuracy under large-scale memory with irrelevant-memory interference.
  • Computational Efficiency: As memory accumulates, baseline computation grows almost exponentially, whereas H-MEM increases slowly and gradually stabilizes.The efficiency advantage becomes increasingly pronounced as memory size grows during continuous task execution.
  • Ablation Study: Removing H-MEM retrieval or both hierarchical storage and retrieval causes clear performance degradation in long-term dialogue tasks.The ablation results indicate that retrieval depends on structured hierarchical storage and that the two components work synergistically.

5 Conclusion and Future Work

The paper concludes that H-MEM organizes and updates long-term memory hierarchically, using layered summarization and positional indexing for structured storage and efficient retrieval. Comparative and efficiency experiments report advantages across long-term dialogue tasks, while future work targets more efficient and multimodal memory mechanisms.

  • Conclusion: H-MEM organizes and updates memory across four semantic-abstraction layers, adding positional indexes for subordinate memories.The architecture uses layer-by-layer summarization to support structured storage and efficient retrieval.
  • Conclusion: Comparative experiments across five long-term dialogue QA tasks and five baselines report significant advantages in preserving and accessing conversational memory.The conclusion also states that efficiency experiments confirm high retrieval efficiency under complex memory conditions.
  • Future Work: Future work will develop more efficient memory mechanisms and extend H-MEM to multimodal memory representations.

6 Limitations

H-MEM has limitations in multimodal support, memory capacity, and privacy and security for long-term dialogue use.

  • H-MEM primarily supports text-based memory, limiting direct processing of images, audio, and video in multimodal dialogues.The architecture cannot directly process non-textual information such as images or videos.
  • H-MEM’s memory capacity may become exhausted as dialogue content grows, while external storage can add latency and management overhead.Increasing capacity also creates memory-lifecycle challenges, including expiration and deletion.
  • Long-term storage of user interactions raises privacy and security concerns involving sensitive data, access control, and possible tampering or theft.The paper identifies the need for mechanisms governing memory access, use, and protection against attackers.
Loading 2507.22925v1…