PaperlayerView exact revision on arXiv

Source-linked AI summary

HAGE: Harnessing Agentic Memory via RL-Driven Weighted Graph Evolution

Dongming Jiang, Yi Li, Guanpeng Li, Qiannan Li, Bingzhe Li

arXiv:2605.09942v1cs.AI

TL;DR

Agentic memory systems struggle to prioritize complex, query-relevant connections for long-horizon reasoning. HAGE addresses this with query-conditioned traversal over weighted relational graphs trained by reinforcement learning, improving reasoning accuracy and efficiency trade-offs.

  • Problem

    Graph-based agent memory still lacks robust ways to prioritize and navigate complex semantic, temporal, causal, and entity-centric connections for long-horizon reuse.

  • Method

    HAGE uses weighted multi-relational graph traversal with query-dependent routing and reinforcement learning to jointly optimize edge representations and retrieval policies.

  • Results

    HAGE improves long-horizon reasoning accuracy and provides a favorable accuracy-efficiency trade-off versus state-of-the-art agentic memory systems.

  • Takeaways & Limitations

    Dynamic, trainable, relation-aware memory structures are a promising foundation for more capable LLM agents.

  • Takeaways & Limitations

    Evaluation covers only LoCoMo and HotpotQA, so results may not generalize to procedural or document-grounded reasoning tasks.

Abstract

from arXiv · show

Memory retrieval in agentic large language model (LLM) systems is often treated as a static lookup problem, relying on flat vector search or fixed binary relational graphs. However, fixed graph structures cannot capture the varying strength, confidence, and query-dependent relevance of relationships between events. In this paper, we propose HAGE, a weighted multi-relational memory framework that reconceptualizes retrieval as sequential, query-conditioned traversal over a unified relational memory graph. Memory is organized as relation-specific graph views over shared memory nodes, where each edge is associated with a trainable relation feature vector encoding multiple relational signals. Given a query, an LLM-based classifier identifies the relational intent, and a routing network dynamically modulates the corresponding dimensions of the edge embedding. Traversal scores are computed via a learned combination of semantic similarity and these query-conditioned edge representations. This allows memory traversal to prioritize high-utility relational paths while softly suppressing noisy or weakly relevant connections. Beyond adaptive traversal, HAGE further introduces a reinforcement learning-based training framework that jointly optimizes routing behavior and edge representations using downstream tasks. Finally, empirical results demonstrate improved long-horizon reasoning accuracy and a favorable accuracy-efficiency trade-off compared to state-of-the-art agentic memory systems. Our code is available at https://github.com/FredJiang0324/HAGE_MVPReview.

1 Introduction

HAGE addresses query-dependent limitations in agentic memory by learning relation-aware traversal over a weighted multi-relational graph. Its reinforcement-learning framework jointly optimizes routing and edge representations using downstream feedback.

  • Motivation: Long-horizon agents need to retain and selectively reuse past evidence, because context-only interaction can dilute, misplace, or forget relevant information.These failures can destabilize recall and degrade long-term reasoning as interactions grow.
  • Motivation: Existing graph memories often encode whether connections exist rather than their query-dependent utility, limiting navigation across semantic, temporal, causal, and entity-centric dependencies.Even continuous weights may remain governed by fixed similarity search, manually designed scores, or static traversal heuristics.
  • HAGE Framework: HAGE reframes retrieval as query-conditioned traversal over relation-specific graph views with trainable edge embeddings and reinforcement-learning-based optimization.This shifts agentic memory from fixed heuristic retrieval toward learned relation-aware retrieval.
  • HAGE Framework: An LLM classifier identifies relational intent, while routing dynamically modulates multi-dimensional edge features and combines structural weight with semantic similarity.The resulting routing can traverse structurally critical but semantically distant bridge nodes.
  • Training: HAGE jointly optimizes routing behavior and edge representations as a sequential decision process using downstream task feedback and node-level evidence targets.The framework does not require full path-level trajectory supervision.
  • Empirical Analysis: Joint optimization with regularization improves generalization over routing-only and edge-only variants, underscoring the importance of learned edge representations.The empirical analysis highlights learned edge representations as important for robust graph-based memory retrieval.

2 Background

Conventional RAG retrieves external information for generation, but long-horizon agents require mutable memory that supports repeated read–generate–write interactions. HAGE addresses static graph access by learning query-conditioned traversal decisions and relation-aware representations through downstream feedback.

  • Retrieval-Augmented Generation: RAG retrieves relevant information from an external datastore and conditions generation on the retrieved context.This paradigm is described as effective for relatively static corpora.
  • Memory-Augmented Generation: Long-horizon agents require memory that accumulates, updates, and reuses information from interactions, including events, preferences, outcomes, and feedback.Memory-Augmented Generation revises the memory store over time as new information is observed.
  • Agentic Memory: Agentic memory uses a read–generate–write loop in which the system retrieves evidence, generates an output, and updates its mutable memory state.This distinguishes agentic memory from conventional retrieval and makes evidence access itself a system responsibility.
  • Learning Structured Retrieval: Graph-based memory access often remains static because systems rely on fixed edge types, manually designed weighting rules, or heuristic traversal procedures.HAGE focuses on learning retrieval behavior in structured memory, including which neighbors to expand and which relational cues to emphasize.
  • HAGE: HAGE formulates graph retrieval as a sequential decision process and trains edge representations and routing behavior using rewards from downstream evidence quality.Edge features encode relation-aware traversal preferences, while the routing policy learns graph traversal under task-level feedback.

3 HAGE Design

HAGE treats agentic memory retrieval as sequential, query-conditioned traversal over a weighted multi-relational graph rather than static lookup. It jointly learns query-aware routing and relation features with reinforcement learning to prioritize structurally useful paths.

  • Framework overview: HAGE combines a weighted multi-relational memory graph with reinforcement learning that jointly optimizes relation-aware retrieval policies and edge representations.The framework makes relation weighting query-adaptive and learnable, unlike fixed edge types and hand-designed scoring rules.
  • Memory graph: Memory is represented as a directed multigraph with four relation-specific edge subsets: temporal adjacency, semantic similarity, causal dependence, and entity co-reference.Event-Nodes include content, timestamps, dense semantic embeddings, and structured metadata.
  • Memory graph: Each edge carries a trainable relation feature vector eij ∈ R^R with R = 4, initialized from cached scores or a one-hot primary relation and updated by downstream rewards.These features encode temporal, semantic, causal, and entity-based relations.
  • Query-conditioned traversal: For each query, QueryRouter combines edge features, relation intent, and semantic similarities into a structural weight that is added to semantic relevance during traversal.The additive score allows structurally important bridge nodes to be selected even when their target-node cosine similarity is negative.
  • Reinforcement learning: HAGE formulates traversal as a Markov Decision Process and trains QueryRouter and edge features jointly with policy gradients, using evidence rewards, traversal-cost penalties, and baseline subtraction.Episodes terminate on target evidence, dead ends, or hop-budget exhaustion; L2 anchor regularization limits drift from Phase 1 feature initialization.

4 Experiments

Experiments evaluate HAGE on long-term conversational and non-conversational multi-hop memory benchmarks, component ablations, and deployment-time accuracy–efficiency trade-offs. HAGE achieves leading performance across settings, while learned edges and routing jointly provide the strongest results.

  • Datasets: HAGE is evaluated on LoCoMo’s ultra-long conversations and HotpotQA’s distractor setting for multi-hop reasoning over dispersed supporting evidence.LoCoMo averages 9K tokens and tests long-range temporal and causal retrieval, while HotpotQA tests evidence combination across distractor passages.
  • LoCoMo Results: 0.739 overall judge score with gpt-4o-mini and 0.548 with Qwen2.5-3B make HAGE the best LoCoMo method under both backbones.These improve on strongest baselines scoring 0.700 and 0.499, respectively.
  • HotpotQA Results: 0.678 F1 and 0.824 LLM score make HAGE the best HotpotQA distractor-setting method with GPT-4o-mini, with the same trend under Qwen2.5-3B.The results support generalization beyond conversational memory to non-conversational multi-hop reasoning.
  • Efficiency: 3.82K tokens per query and 2.17s average latency accompany HAGE’s highest average score, yielding a favorable accuracy–efficiency trade-off.Token consumption and latency remain within the same order of magnitude as other retrieval-based memory methods.
  • Ablation: 0.739 Judge and 0.548 F1 are achieved by full HAGE, exceeding static edges at 0.698, LLM-scored edges at 0.712, and trainable edges at 0.724.The ablation indicates that learned edge representations and trainable routing are complementary.

5 Conclusion

HAGE formulates agentic memory retrieval as query-conditioned traversal over dynamic relational graphs within a weighted multi-relational memory framework. It combines relation-aware traversal with reinforcement-learning optimization to adapt retrieval to query intent and downstream task feedback, improving long-horizon reasoning accuracy and accuracy-efficiency trade-offs.

  • HAGE presents a weighted multi-relational memory framework that formulates agentic memory retrieval as query-conditioned traversal over dynamic relational graphs.
  • HAGE couples relation-aware graph traversal with reinforcement-learning optimization of routing policies and edge representations.This enables retrieval to adapt to both query intent and downstream task feedback.
  • Empirical results show improved long-horizon reasoning accuracy and a favorable accuracy-efficiency trade-off compared to state-of-the-art agentic memory systems.

Limitations

HAGE’s results are scoped by limited benchmark coverage and dependence on instruction-tuned LLM components. These constraints may limit generalizability and introduce cost, model-specific variability, and sensitivity to relation-intent classification quality.

  • Benchmark coverage: Evaluation on only LoCoMo and HotpotQA may not generalize fully to procedural or document-grounded reasoning tasks.The benchmarks cover long-term conversational memory and non-conversational multi-hop question answering, respectively.
  • Dependence on LLM components: Query analysis and LLM-as-a-Judge evaluation rely on instruction-tuned LLMs, introducing cost and model-specific variability.Relation intent extraction and evaluation both depend on these components.
  • Dependence on LLM components: Relation-intent classifier accuracy directly affects the quality of query-conditioned edge features used during traversal.Errors in relation intent extraction can therefore influence traversal behavior.

Ethical Considerations · A Related Work

The paper notes privacy, consent, and bias risks in persistent, reinforcement-learning-based memory retrieval while using publicly available licensed datasets and models. It situates HAGE among context-extension, retrieval-augmented, structured graph-memory, and learned memory-access research, combining multi-relational graph memory with query-conditioned RL routing.

  • Ethical Considerations: Persistent memory can retain sensitive interaction histories beyond their intended scope, creating privacy and consent risks if stores are accessed without safeguards.RL-optimized retrieval policies may also surface information in ways reflecting biases in training data.
  • Ethical Considerations: All datasets and models are publicly available and used under their respective licenses, with no new datasets introduced.The listed resources include LoCoMo, HotpotQA, all-MiniLM-L6-v2, GPT-4o-mini, and Qwen2.5-3B.
  • A Related Work: Related work is organized along four axes: context-window extension, retrieval-augmented generation, structured and graph-based agent memory, and learning memory access policies.These axes position HAGE within broader surveys and research on agentic memory.
  • A Related Work: Context-window extension methods modify attention or positional encodings, add auxiliary memory modules, or enhance retrieval with global memory to handle inputs exceeding extended windows.These approaches mitigate context-length bottlenecks but do not address the complete limitation described in the truncated passage.
  • A Related Work: RAG conditions generation on passages retrieved from external corpora, with later work extending long-context retrieval, multi-partition retrieval, and retrieval serving.Classical formulations generally assume relatively static knowledge bases and externally provided documents.
  • A Related Work: Structured memory systems support richer reasoning through hierarchies, selective episodic write-back, Zettelkasten-style links, and other representations beyond flat vector stores.Examples include MemGPT, MemoryBank, Nemori, and A-MEM.
  • A Related Work: Graph-based architectures explicitly encode relational structure through entity-centric community graphs, evolving episodic world models, and hierarchical Event-Node and Episode-Node organization.These approaches target relational reasoning and long-horizon retrieval for LLM agents.
  • A Related Work: Learned memory-access research frames retrieval as a decision process, using reinforcement learning or iterative access to optimize what and when to retrieve.AgeMem demonstrates that downstream reward signals can guide memory operations.

B Implementation Details

HAGE represents each sample graph with MiniLM node embeddings, COO edge indices, edge-type labels, and four-dimensional edge features. Training uses sample-level 5-fold cross-validation with fixed validation and test partitions, Adam optimization, and specified routing and edge learning rates.

  • Training Protocol: Training uses sample-level 5-fold cross-validation, with 20% validation data and 10% strictly reserved as an unseen test set per fold.Each fold trains for 200 epochs with Adam, using ηrouter = 10−3 and ηedge = 10−4.
  • Reinforcement Learning Hyperparameters: The training configuration uses γ = 0.99, β = 0.99, λanchor = 1.0, Hmax = 5, Rhit = 10.0, λstep = 0.05, and λtimeout = 1.0.These specify discounting, baseline decay, anchor regularization, hop budget, hit reward, step penalty, and timeout penalty, respectively.

C Prompt Library

HAGE’s prompt library uses distinct prompts for structured event extraction, query-adaptive answer generation, and semantic LLM-based evaluation. Together, these prompts convert conversations into graph-compatible memory, condition answering on query type and retrieved context, and assess semantic fidelity beyond n-gram overlap.

  • C Prompt Library: HAGE uses three prompt types, each optimized for a specific cognitive task in the memory pipeline.The library covers event extraction, query-adaptive question answering, and LLM-as-a-judge evaluation.
  • C.1 Structured Event Extraction Prompt: The event extraction prompt converts conversational turns into structured memory units containing entities, topics, relationships, temporal cues, and factual summaries.Its lightweight structured output is directly consumed by the memory construction pipeline.
  • C.1 Structured Event Extraction Prompt: The extraction template accepts speaker, text, and optional prior-summary context, then returns entity, topic, relationship, fact, temporal, and attributed-summary fields.This schema replaces free-form generation with explicitly defined output fields.
  • C.2 Query-Adaptive QA Prompt: The QA prompt supplies retrieved memory context with the user query and adapts generation according to the query type predicted by the router.This provides lightweight query-specific reasoning control while keeping answers grounded in retrieved memory.
  • C.2 Query-Adaptive QA Prompt: The query-adaptive QA template requires contextual grounding, concise factual answers when appropriate, abstention when unsupported, and compliance with router-generated guidance.The prompt inputs are retrieved context, the user question, and a router-generated instruction.
  • C.3 Evaluation Prompt (LLM-as-a-Judge): The evaluation prompt uses a Judge LLM to score semantic alignment between generated responses and ground truth on a continuous scale [0.0, 1.0].Its rubric evaluates entities, temporal markers, causal relationships, correctness, and missing constraints rather than relying only on n-gram overlap.

D Baseline Configurations

The evaluation standardizes the experimental environment across systems, using a full-context reference, official baseline settings, a shared gpt-4o-mini backbone, and identical judging. It evaluates all five LoCoMo cognitive categories.

  • Evaluation Protocols: Experiments standardized the environment across all systems to ensure fair and rigorous comparison.The protocols define the comparison setup.
  • Evaluation Protocols: The Full Context baseline feeds conversation history directly into gpt-4o-mini, up to its 128k token limit, without external retrieval.It measures native long-context capabilities as a brute-force reference.
  • Evaluation Protocols: Retrieval-based baselines, including AMem, Nemori, and MemoryOS, use their official default hyperparameters and storage settings.This reflects standard out-of-the-box performance.
  • Evaluation Protocols: All systems use OpenAI’s gpt-4o-mini for retrieval reasoning and response generation, while outputs share an identical gpt-4o-mini judge at temperature=0.0.The unified backbone controls foundation-model variance, and unified evaluation applies the same judging framework.
  • Dataset Statistics: Evaluation covers the full LoCoMo benchmark across all five cognitive categories to assess varying retrieval complexity.The detailed query-type distribution is reported in Table 5.

E Dataset and Model Licenses

The experiments use publicly available datasets and models under their respective licenses or terms of service. All datasets are used for research purposes, and no new datasets are introduced.

  • LoCoMo is released under CC BY-NC 4.0, while HotpotQA is released under CC BY-SA 4.0.
  • all-MiniLM-L6-v2 is released under Apache License 2.0.
  • GPT-4o-mini is accessed via the OpenAI API under OpenAI’s Terms of Service.
  • Qwen2.5-3B is released under the Qwen Research License Agreement.
  • All datasets are used for research purposes consistent with their respective licenses, and no new datasets are introduced.
Loading 2605.09942v1…