GAM: Hierarchical Graph-based Agentic Memory for LLM Agents

Zhaofen Wu; Hanrong Zhang; Fulin Lin; Wujiang Xu; Xinran Xu; Yankai Chen; Henry Peng Zou; Shaowen Chen; Weizhi Zhang; Xue Liu; Philip S. Yu; Hongwei Wang

Source-linked AI summary

GAM: Hierarchical Graph-based Agentic Memory for LLM Agents

Zhaofen Wu, Hanrong Zhang, Fulin Lin, Wujiang Xu, Xinran Xu, Yankai Chen, Henry Peng Zou, Shaowen Chen, Weizhi Zhang, Xue Liu, Philip S. Yu, Hongwei Wang

arXiv:2604.12285v1Apr 14, 2026cs.AI

TL;DR

LLM agents need long-term memory that captures new interactions without corrupting established knowledge, yet existing stream and structured architectures expose a stability–adaptability gap. GAM separates episodic buffering from semantic consolidation through hierarchical graphs and semantic-event-triggered updates, with graph-guided retrieval. Across LoCoMo and LongDialQA, the paper reports superior reasoning accuracy and efficiency over strong baselines.

Problem
LLM agents must rapidly capture real-time interactions while preserving established knowledge from transient noise, a conflict existing architectures do not fully resolve.
Method
GAM decouples encoding from consolidation using local Event Progression Graphs, a global Topic Associative Network, semantic-event-triggered state switching, and graph-guided multi-factor retrieval.
Results
GAM consistently outperforms state-of-the-art baselines in reasoning accuracy and computational efficiency across LoCoMo and LongDialQA.
Takeaways & Limitations
State-based separation and multi-signal graph retrieval provide GAM with a memory architecture for preserving long-term consistency while handling evolving dialogue.
Takeaways & Limitations
The current GAM instantiation is limited to text modality and cannot perform fine-grained reasoning over visual or auditory inputs.

Abstract

from arXiv · show

To sustain coherent long-term interactions, Large Language Model (LLM) agents must navigate the tension between acquiring new information and retaining prior knowledge. Current unified stream-based memory systems facilitate context updates but remain vulnerable to interference from transient noise. Conversely, discrete structured memory architectures provide robust knowledge retention but often struggle to adapt to evolving narratives. To address this, we propose GAM, a hierarchical Graph-based Agentic Memory framework that explicitly decouples memory encoding from consolidation to effectively resolve the conflict between rapid context perception and stable knowledge retention. By isolating ongoing dialogue in an event progression graph and integrating it into a topic associative network only upon semantic shifts, our approach minimizes interference while preserving long-term consistency. Additionally, we introduce a graph-guided, multi-factor retrieval strategy to enhance context precision. Experiments on LoCoMo and LongDialQA indicate that our method consistently outperforms state-of-the-art baselines in both reasoning accuracy and efficiency.

1 Introduction

GAM addresses the tension between rapidly capturing dialogue and preserving stable knowledge by decoupling episodic encoding from semantic consolidation. Its hierarchical graphs, state-based updates, and graph-guided retrieval reduce interference while improving reasoning accuracy and efficiency.

LLM agents must capture real-time interactions while protecting established knowledge from transient noise corruption.
Unified stream-based systems risk memory contamination, memory loss, and semantic drift through direct updates to active memory.
GAM separates episodic buffering from semantic consolidation, integrating local event details into global memory only after semantic shifts.
State-based consolidation transitions between Episodic Buffering and Semantic Consolidation states using semantic divergence rather than arbitrary triggers.
Graph-guided retrieval combines temporal, confidence, and role-based signals in top-down traversal to recover precise episodic details from semantic themes.
Evaluations on LoCoMo and LongDialQA show higher reasoning accuracy and computational efficiency than state-of-the-art baselines.

2 Related Work

Prior memory systems trade off rapid updating against stable, structured retention. GAM is positioned as an explicit separation of the memory lifecycle rather than an arbitrary state mechanism or retrieval-only optimization.

Unified stream systems record observations continuously and use retrieval or hierarchical tiers to manage expanding context.
Some dynamic systems trigger updates by token counts or time steps, which do not ensure semantic completeness.
Discrete structured architectures support precise multi-hop reasoning but can become rigid, latent, and expensive to index.
Dynamic structured approaches add episodic nodes or optimize retrieval infrastructure, but may rely on artificial state definitions or retrieval-only improvements.
GAM explicitly decouples encoding and consolidation to balance immediate accessibility against long-term interference.

3 Methodology

GAM models memory management as an online decision process that separates rapid episodic encoding from stable semantic consolidation. Its hierarchical graphs, semantic boundary detection, and graph-guided retrieval preserve transient details while connecting them to precise long-term context.

Hierarchical Graph Memory Architecture: GAM balances immediate accessibility against long-term interference by separating the local Event Progression Graph from the global Topic Associative Network.The active event graph supports rapid perception, while the topic network provides the stable global substrate; archived graphs and cross-layer links preserve grounded evidence.
State-Based Memory Consolidation: Semantic divergence detection triggers consolidation when local context has drifted sufficiently from the global state, replacing arbitrary update timing with a low-cost structural-shift policy.The discriminator is invoked at sparse maintenance events, including session-end markers, natural pauses, or buffer overflow, rather than every turn.
State-Based Memory Consolidation: During episodic buffering, incoming utterances are appended to an isolated local graph, protecting the global network from temporary noise and reducing encoding latency.This write-isolation design separates acquiring new information from modifying long-term memory.
Hierarchical Graph Memory Architecture: The composite memory structure contains a stable Topic Associative Network, a rapidly updated Event Progression Graph, and archived event graphs linked through cross-layer associations.Topic nodes represent high-level themes and semantic correlations, whereas event nodes represent real-time interaction units and their temporal or causal evolution.
State-Based Memory Consolidation: At a detected topic boundary, consolidation merges the buffered event graph into a semantic node containing both an LLM-generated summary and concatenated raw text.The summary supports thematic reasoning, while raw content preserves fine-grained details and limits information loss.
Graph-Guided Multi-Factor Retrieval: Graph-guided retrieval expands top semantic anchors through first-order topic neighbors, drills down through cross-layer links to archived event graphs, and integrates temporal, confidence, and role signals.This traversal bridges separated storage layers to retrieve context that is semantically deep and temporally precise.

4 Experiments

Experiments on LoCoMo and LongDialQA evaluate GAM against diverse memory architectures, showing strong reasoning performance and efficiency alongside component and partitioning analyses.

Experimental Settings: LoCoMo evaluates GAM across four LLM backbones using F1 and BLEU-1, against heuristic, stream-based, and self-evolving memory baselines.The evaluation includes MemoryBank, ReadAgent, MemGPT, MemoryOS, Mem0, and A-Mem.
LoCoMo Results: GAM achieves the highest Average F1 of 43.14 on GPT-4o-mini, while Mem0 remains stronger on Temporal F1, 56.42 versus 51.96.Open-Domain is consistently the most difficult category for GAM.
LongDialQA Results: On LongDialQA, GAM reaches an Average F1 of 12.55 with Qwen 2.5-7B, surpassing MemoryOS by 86%.The reported advantage holds across all sub-datasets and is attributed to reduced interference from frequent speaker switching.
Ablation Study: The full method consistently outperforms ablated variants; removing EPG causes the largest degradation, while removing SSM, TAN, or MFR also reduces performance.These results support the contributions of narrative structure, encoding-consolidation decoupling, topic organization, and multi-signal retrieval.
Partitioning Analysis: GAM records the highest average F1 of 40.00 under semantic-boundary partitioning, compared with 34.23 for Fixed Window (256).On Temporal tasks, GAM improves over the best baseline from 44.12 to 48.97 and exceeds Session-based partitioning at 36.59.
Efficiency Analysis: GAM uses 1,370 tokens per query, reducing token consumption by 11% versus Mem0, while delivering a 13% F1 gain at comparable speeds.Mem0 has slightly lower latency, whereas GAM outperforms A-Mem and MemoryOS on both reported metrics.

5 Conclusion

GAM addresses the tension between rapid context perception and stable knowledge retention by separating narrative buffering from semantic consolidation. Its graph-guided retrieval combines temporal, confidence, and role-centric signals, and the reported evaluations show superior reasoning accuracy and efficiency over strong baselines.

Conclusion: GAM separates narrative buffering from semantic consolidation through state-based switching, updating memory only at semantically complete boundaries.This design isolates transient noise while preserving established knowledge.
Conclusion: Graph-guided retrieval fuses temporal, confidence, and role-centric signals to support precise recall across the decoupled memory layers.The framework is described as a hierarchical graph-based agentic memory architecture.
Conclusion: GAM achieves superior reasoning accuracy and efficiency over strong baselines.The paper identifies native multimodal memory as future work for textual, visual, and auditory signals.

Limitations

GAM’s current implementation is limited to text, excluding fine-grained reasoning over visual and auditory inputs. The authors leave native multimodal extension to future work.

GAM currently supports only text modality, limiting access to nonverbal visual and acoustic information.The authors identify native multimodal memory nodes as a future extension.
The current system cannot perform fine-grained reasoning over visual or auditory inputs.
A future extension would combine symbolic summaries with modality-specific features such as keyframe or audio representations.

Ethical Considerations

GAM’s graph structure creates intervention points for privacy filtering, user-directed memory control, and local correction of inaccurate beliefs. These affordances remain architectural possibilities rather than complete safety solutions.

Privacy: Before buffered content enters the global topic graph, privacy filters or retention rules can inspect and block sensitive information.
User control: Explicit graph nodes and cross-layer links make inspection, selective deletion, freezing, and branch-level pruning easier to implement.The passage states that deployment need not expose these controls automatically.
Error correction: Faulty summaries or semantic links can in principle be corrected locally by removing edges, adjusting confidence weights, or deleting incorrect topic nodes.The authors characterize these as safety primitives requiring interface design, auditing, and user consent.

A Implementation Details

GAM maintains a bounded episodic buffer and invokes semantic-boundary detection sparsely rather than scanning every incoming turn. Consolidation then narrows topic-linking candidates before applying more expensive relation scoring.

Workflow: The workflow summarizes GAM’s online maintenance and inference operations.
Online maintenance: GAM maintains a bounded episodic buffer and invokes the boundary discriminator only during sparse maintenance events.
Consolidation: The consolidation stage first narrows candidate topic nodes through vector retrieval before applying LLM-based relation scoring.This avoids scoring a new semantic node against every existing topic node.

B.1 Detailed Analysis of LongDialQA Results

GAM’s LongDialQA analysis evaluates graph-guided retrieval and event-driven memory maintenance across multi-party dialogue benchmarks and multiple backbones. The reported results show strong performance, stable scaling behavior, sparse maintenance overhead, and gains in a visual-caption proxy setting.

Memory Maintenance: The online workflow updates the event graph for each utterance, computes a boundary indicator, generates a topic node, updates the topic graph, and establishes cross-layer links.
Graph-Guided Retrieval: The inference workflow identifies topic anchors, expands them semantically, extracts candidates, and returns top-ranked context for generation.
Semantic Boundary Detection: The event-driven detector checks session or token-limit conditions, estimates a boundary indicator, triggers consolidation, and then resets or trims the local buffer.
B.1 Detailed Analysis of LongDialQA Results: Table 5 compares F1 and BLEU-1 across Big Bang Theory, Friends, The Office, and their average, with best and second-best results marked per backbone.
LongDialQA Results: 11.60 vs 9.11: GAM surpasses Mem0 on The Office using the Qwen 2.5-7B backbone, while the reported advantage is associated with role-centric contextualization in dense multi-speaker narratives.
Backbone Robustness: 7.36: GAM achieves the highest Average F1 with Llama 3.2-3B, maintaining an advantage where A-Mem and MemoryOS fluctuate across backbone sizes.

C.3.1 Experimental Setup

The evaluation tests GAM under topic-segmentation noise and across ablations, baselines, multimodal inputs, and consolidation-cost measurements. The setup includes miss, shift, and extra boundary errors with noise levels up to η = 0.4.

Robustness setup: The robustness experiment injects miss, shift, and extra segmentation errors while varying noise level η from 0.0 to 0.4.Miss errors remove ground-truth boundaries, shift errors move them by up to two turns, and extra errors insert random cut-points.
Evaluation coverage: The reported experiments include ablations, comparison with AriGraph, proxy multimodal evaluation on MovieChat-1K, and complexity analysis of progressive session injection.The ablation table covers LoCoMo task categories, while the multimodal evaluation uses dense video captions as input.
Evaluation coverage: The study also measures the cost of semantic divergence detection on LoCoMo.This complements performance evaluations with an efficiency-oriented analysis of the consolidation trigger.
Robustness setup: The evaluation compares GAM against Fixed Window and Session-based baselines under segmentation noise.Figure 4 reports Average F1 and task-category performance as noise increases.

C.3.2 Results and Analysis

GAM remains robust under severe segmentation noise, maintaining strong overall performance and outperforming the Fixed Window and Session-based baselines at η = 0.4. Task-level effects vary: temporal retrieval can benefit from over-segmentation, while single-hop retrieval degrades only modestly.

Overall robustness: Average F1 remains consistent around 40.0 despite boundary deletions and shifts.The result indicates stable performance under complex segmentation perturbations.
Overall robustness: 38.60 F1 at η = 0.4 outperforms Fixed Window at 34.23 and Session-based at 36.59.This is the most severe tested segmentation-noise level.
Task-level analysis: Temporal tasks peak at 51.72 when η = 0.3, while single-hop retrieval declines from 54.58 to 51.98.Extra boundaries may create finer-grained nodes for temporal grounding, whereas boundary shifts or deletions can disrupt local keyword-matching coherence.
Task-level analysis: Single-hop degradation remains below 5% as graph-guided multi-factor retrieval bridges gaps from segmentation errors.The reported explanation attributes this compensation to the graph’s semantic connectivity.

D.1 Modulation Factors Sensitivity Analysis

The sensitivity analyses examine modulation factors, retrieval size, prompting, graph-edge construction, representation generation, and topic-boundary detection. Results show task-dependent factor trade-offs, stable confidence-factor behavior, and a preferred retrieval size of k = 10.

Modulation factors: Confidence-factor curves remain relatively flat from 1.0 to 2.0, with a slight peak between 1.2 and 1.4.The analysis reports high stability for βconf and a larger role for graph topology than precise tuning.
Modulation factors: Role-factor performance peaks at approximately βrole = 1.4, especially for Multi-hop tasks, but declines slightly when β > 1.6.Speaker-identity filtering helps connect character-associated information, while excessive filtering may exclude relevant context from other speakers.
Modulation factors: Temporal reasoning is optimal around βtime = 1.4, whereas Open-domain performance dips at that value.The reported trade-off is between sharper temporal focus and a narrower retrieval scope for broad questions.
Overall sensitivity: Low Average F1 standard deviation across parameter ranges indicates robustness to hyperparameter variation.The analysis attributes effectiveness primarily to architectural design rather than overoptimized hyperparameters.
Retrieval size: Performance peaks at retrieval size k = 10 and then stagnates or degrades as k increases to 40.Larger retrieval sets can introduce irrelevant noise; Temporal tasks are particularly sensitive, while Multi-hop tasks remain more robust.
Graph construction: The global Topic Associative Network assigns edge types and LLM-derived confidence weights to prioritize high-confidence associations.The prompt supports relation labels including support, contradiction, coreference, causal, semantic, and unrelated.
Representation and detection: Semantic consolidation generates keywords and concise summaries for lexical matching and semantic embedding retrieval.The associated structured prompt requests salient keywords and a short summary in JSON format.