Source-linked AI summary
Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation
Zhanghao Hu, Qinglin Zhu, Runcong Zhao, Di Liang, Hanqi Yan, Yulan He, Lin Gui
TL;DR
Agent memory’s coherent, highly overlapping histories make flat similarity retrieval redundant, while progressive compression can blur distinguishing details. xMemory decouples histories into evidence components, aggregates them into a revisable hierarchy, and retrieves top-down. Across LoCoMo and PerLTQA, it improves answer quality and inference token efficiency, while its construction cost and LLM-based extraction remain practical limitations.
Problem
Agent memory differs from heterogeneous RAG corpora because correlated histories make fixed similarity retrieval redundant and compressed hierarchies can blur distinguishing details.
Method
xMemory segments histories, decouples segments into memory components, aggregates components into groups, revises the hierarchy, and retrieves top-down with selective text expansion.
Results
Across LoCoMo and PerLTQA, xMemory improves answer quality and inference token efficiency.
Takeaways & Limitations
Effective agent memory can reorganise past interactions into evidence-oriented structures rather than only retrieving or compressing them.
Takeaways & Limitations
xMemory adds memory-construction cost, and errors in LLM-generated component extraction may affect downstream retrieval.
Abstract
from arXiv · showhide
Standard Retrieval Augmented Generation (RAG) is poorly matched to agent memory. Unlike large heterogeneous corpora, agent memory forms a bounded and coherent interaction stream in which many spans are highly correlated or near duplicates. As a result, flat top-$k$ similarity retrieval often returns redundant context, while summary-centric hierarchies can blur the subtle details that distinguish one candidate from another. We argue that agent memory should follow the principle of decoupling before aggregation: the system should first isolate reusable facts, updates, and distinguishing details from similar histories, and only then organise them for efficient retrieval. Based on this principle, we propose xMemory, which constructs a revisable hierarchical memory structure from original messages to segments, memory components, and groups. xMemory segments interaction history into local events, decouples each segment into memory components, aggregates related components into high-level groups using a sparsity--semantic faithfulness objective, and maintains this structure incrementally as memory evolves. At inference time, xMemory retrieves top-down, first selecting a compact backbone of complementary groups and components, and then expanding to segments and raw messages only when additional evidence reduces the reader's uncertainty. Experiments on LoCoMo and PerLTQA across diverse open source and closed source LLMs show consistent gains in answer quality and inference token efficiency, supported by analyses of redundancy, evidence density, and coverage.
1 Introduction
Agent memory differs from standard RAG because coherent, overlapping histories make similarity retrieval redundant and compression can blur answer-critical distinctions. xMemory responds by decoupling evidence before aggregation, maintaining a revisable hierarchy and retrieving it adaptively.
- Coherent, overlapping agent-memory streams make fixed top-k similarity retrieval overemphasize redundant evidence.
- Progressively compressed hierarchies can preserve shared background while blurring details that distinguish candidates.
- xMemory decouples histories into reusable facts, updates, and distinctive components before aggregating them into a revisable hierarchy.
- xMemory retrieves top-down, selecting complementary groups and components before expanding to segments and messages when more evidence is needed.
- The paper reports improved answer quality and inference token efficiency on LoCoMo and PerLTQA.
2 Related Work
Existing agent-memory systems commonly adapt RAG retrieval or hierarchical organisation, but similarity-based and summary-centric designs can leave redundancy and fine-grained distinctions unresolved. Dynamic insertion also differs from retroactive reorganisation as later interactions reshape the interpretation of earlier evidence.
- RAG-style memory systems retrieve stored interactions by similarity, which can return redundant spans while missing small updates or temporal distinctions.
- Summary-, note-, and graph-based hierarchies improve scalability but may abstract away fine-grained distinctions at higher levels.
- Incrementally inserting new memories into existing structures is distinct from retroactively reorganising earlier evidence as later interactions change its relations.
3 Method
xMemory builds a revisable hierarchy by segmenting interaction history, decoupling segments into reusable evidence components, and aggregating components into groups. Its top-down retrieval selects complementary high-level evidence, then expands text only when it reduces uncertainty.
- From messages to components: Raw messages are partitioned into coherent local-event segments, which are decoupled into reusable facts, constraints, attributes, relations, and state updates.
- From components to groups: Components are aggregated into compact groups using an objective that balances sparsity with semantic faithfulness.
- Dynamic maintenance: Group maintenance attaches new components to compatible groups and considers splitting incoherent groups or merging small, isolated ones.
- Retrieval: The hierarchy separates high-level evidence units from segments and messages that preserve intact local context.
- Stage I: high-level backbone selection: Stage I greedily selects groups and components by trading structural coverage against query relevance.
- Stage II: adaptive text expansion: Stage II expands to segments or messages only when their additional text yields positive uncertainty reduction, stopping otherwise.
4 Experiments
Experiments on LoCoMo and PerLTQA evaluate xMemory against flat retrieval, compression, and structured-memory baselines across multiple reader models. xMemory achieves the strongest reported answer quality while using competitive or fewer inference tokens.
- Experimental Setup: The evaluation compares full-context reading, flat similarity retrieval, post-retrieval compression, and structured-memory designs on LoCoMo and PerLTQA.LoCoMo contains 50 multi-session dialogues averaging approximately 18K tokens and 300 turns; PerLTQA averages around 25K tokens.
- LoCoMo: Across three reader models on LoCoMo, xMemory achieves the best average performance, including 34.48/43.98 BLEU/F1 with Qwen3-8B.With GPT-5 nano, xMemory improves over Nemori from 36.65/48.17 to 38.71/50.00 while reducing token usage from 9155 to 6581 per query.
- Method Comparison: Compared with baselines, xMemory explicitly decouples distinctive evidence into memory components, organises it into revisable groups, and retrieves top-down.This design is contrasted with redundant raw-context retrieval and post-retrieval compression.
- PerLTQA: xMemory remains consistently effective on PerLTQA across all three backbones, reaching 36.24/47.08/42.50 BLEU/F1/ROUGE-L with Qwen3-8B while using the fewest tokens.With Llama-3.1-8B-Instruct, it reaches 42.68/52.37/47.84.
- Efficiency: On LoCoMo with Qwen3-8B, xMemory reduces token usage from 7755 for Nemori to 4711 while improving average BLEU/F1 from 31.44/40.88 to 34.48/43.98.Token usage includes retrieval, answer generation, and auxiliary calls.
5 Analysis
Ablations show that memory structuring, staged retrieval, suitable group size, and retroactive restructuring jointly improve quality and efficiency. Evidence-density and cost analyses associate these gains with concentrating answer-bearing information into fewer retrieved blocks and tokens.
- Retrieval-Stage Ablation: Memory-only retrieval raises average BLEU/F1 from 27.95/36.48 for Naive RAG to 31.81/40.77, while adding both retrieval stages gives the best trade-off.Stage I selects groups and components, while Stage II expands to segments and messages when uncertainty warrants more evidence.
- Group Size: 34.48 BLEU and 43.98 F1 are achieved with an upper bound of 12 components per group and 4.48 components per group on average.Larger groups weaken discrimination, whereas smaller groups fragment the hierarchy.
- Retroactive Restructuring: 44.91% reassignment and 43.98 average F1 are achieved by the full system, compared with 0% reassignment and 38.59 F1 when split and merge are disabled.Split enables repartitioning broad groups, while merge limits unnecessary fragmentation.
- Evidence Density: On multi-hop questions, xMemory raises 2-hit and multi-hit retrieved-block proportions to 13.14% and 12.19%, versus 7.82% and 6.53% for Naive RAG.Higher-hit blocks contain multiple distinct answer evidence units and indicate denser retrieval.
- Coverage Efficiency: xMemory achieves the best BLEU/F1 while covering answer evidence with substantially fewer blocks and tokens than the comparison settings.Pruning lowers token cost relative to Naive RAG but requires more blocks and yields lower accuracy, consistent with fragmented evidence.
- Cost–Performance Trade-off: xMemory attains the highest average BLEU at lower total token cost than Nemori, MemoryOS, and A-Mem after accounting for construction and inference tokens.The comparison uses total cost as average construction cost plus average inference tokens per query multiplied by query count.
6 Conclusion
The paper concludes that xMemory addresses the mismatch between agent memory and standard RAG by reorganising correlated interactions into evidence-oriented hierarchical structures. Across LoCoMo and PerLTQA, this approach improves answer quality, token cost, and evidence density relative to RAG baselines.
- Conclusion: xMemory disentangles highly correlated memories into semantic components and retrieves top-down over a hierarchical structure.The framework is intended to reduce redundancy while preserving temporally linked evidence.
- Conclusion: Across LoCoMo and PerLTQA, xMemory improves answer quality with lower token cost and retrieves more evidence-dense contexts than RAG baselines.The conclusion frames evidence-oriented reorganisation as a useful direction for long-horizon agent memory.
A Additional Method Details
xMemory incrementally maintains a hierarchical memory structure by segmenting interaction streams, extracting fine-grained components, grouping them adaptively, and expanding evidence only when it reduces reader uncertainty.
- Segmentation: Interaction histories are incrementally partitioned into contiguous segments representing coherent local events.Segmentation uses local semantic continuity, including topical and temporal continuity and discourse structure.
- Component extraction: Each segment is decomposed into concise components representing reusable facts, constraints, attributes, preferences, relations, or state updates.Components are intended to be minimal, self-contained evidence units that avoid unnecessary duplication.
- Component representation: Components retain textual descriptions, embeddings, and source-segment pointers, combining fine-grained retrieval with access to original local context.
- Dynamic maintenance: New components attach to their nearest group when similarity exceeds τattach; otherwise, xMemory creates a new group.The threshold controls whether the existing organisation is sufficiently expressive for incoming evidence.
- Dynamic maintenance: Periodic or threshold-triggered split, merge, and local kNN updates make the hierarchy revisable while confining maintenance to affected regions.Attach is inexpensive, whereas split and merge occur less frequently and avoid full reconstruction.
- Adaptive retrieval: Retrieval selects complementary groups and components top-down, then expands to linked segments and messages only when they reduce reader uncertainty.Uncertainty is represented by predictive entropy when token-level distributions are available, or estimated with a proxy model for black-box readers.
B Implementation Setup
The implementation fixes structure-update defaults, evaluates their sensitivity on LoCoMo, and documents the research-only assets and licenses used in the experiments.
- Hyperparameters: xMemory uses a group size cap of 12, attachment threshold 0.6, post-split clustering threshold 0.65, and merge threshold 0.65 by default.These settings are applied across datasets and backbone models unless otherwise specified.
- Sensitivity analysis: The sensitivity study varies one structure-update threshold at a time while keeping the others fixed on LoCoMo with Qwen3-8B.Results are reported as BLEU/F1, with defaults marked in the table.
- Assets and licenses: The evaluation uses existing datasets, models, and baseline methods under their stated research-use licensing terms.PerLTQA is identified as CC BY-NC 4.0, while LoCoMo follows its authors’ accompanying terms.
C.1 Detailed Ablation Results
The ablations indicate that memory structuring provides a stronger retrieval basis than Naive RAG, while complementary selection and uncertainty-guided expansion improve efficiency and answer quality in different ways.
- Ablation overview: Memory structuring alone provides a substantially stronger retrieval basis than Naive RAG on LoCoMo with Qwen3-8B.
- Stage I: Stage I improves high-level evidence selection while reducing inference token usage.Stage I adds group and component selection to the structured memory.
- Stage II: Stage II contributes stronger answer-quality gains through uncertainty-guided expansion over segments and messages.Stage II adds uncertainty-based segment and message expansion.
- Routing motivation: In high-similarity dialogue streams, bounded discriminative information makes multi-way routing increasingly error-prone as candidate sets grow.The analysis links near-duplicate candidates and compressed representations to weak routing signals.
- Routing motivation: Balancing group sizes limits within-group candidate sets, addressing the regime where bounded discriminative information makes routing errors difficult to avoid.The sparsity objective controls typical within-group candidate size and prevents extremely large groups.
C.2.3 Fano-style Lower Bound and optimal nk in hierarchical structure.
The Fano-style analysis motivates controlling group size under bounded discriminability, and experiments show that a cap of 12 gives the strongest reported LoCoMo trade-off between routing and coverage.
- Lower bound: Fano’s inequality formalizes a lower bound on routing error when observable evidence has limited information about the correct candidate.The bound is applied to component selection within groups.
- Lower bound: Under bounded discriminative information, increasing the within-group candidate count raises the lower bound on misrouting probability.The result motivates limiting group arity in highly similar dialogue memory.
- Optimal group size: A group-size cap of 12 achieves the best BLEU/F1 in every question category and the best average of 34.48/43.98 on LoCoMo with Qwen3-8B.
- Optimal group size: Caps above 12 reduce performance through weaker routing discriminability, whereas caps below 12 over-fragment the hierarchy and weaken multi-fact coverage.
- Optimal group size: The cap of 12 yields an actual average branching factor of approximately 4.5 components per group rather than forcing every group to reach the ceiling.The sparsity and semantic-faithfulness objective determines finer-grained group sizing within the cap.
- Evidence density and coverage: xMemory improves answer quality and coverage efficiency over the compared retrieval settings, requiring 5.66 blocks and 974.56 coverage tokens versus 10.81 and 1979.26 for one comparator.The reported LoCoMo analysis also finds that pruning can discard answer-bearing details, while xMemory improves BLEU and F1 across categories.
- Retroactive restructuring: Allowing split or merge enables retroactive reassignment and improves QA, while the full system achieves the highest reassignment ratio and best accuracy across categories.Split contributes more reassignment, while merge consolidates redundant groups into a compact organisation.
- Cost–performance trade-off: xMemory adds moderate construction overhead but substantially reduces inference-time token cost while maintaining the reported performance comparison.The cost–performance analysis uses average construction and inference-time token statistics on LoCoMo with Qwen3-8B.
D Case Study: Decoupling before Aggregation
The LoCoMo case study shows that flat similarity retrieval mixes a decisive job-loss event with later related events, whereas xMemory separates these facts and preserves the source context needed to answer temporally.
- The query asks when Gina lost her DoorDash job, with the gold answer being January 2023.
- Flat retrieval selects broadly relevant memories but mixes the original job-loss event with later business-transition consequences, producing a vague answer.
- xMemory uses a group for coarse access, components to distinguish related facts, and a source-linked segment containing the January temporal evidence.
- The hierarchy preserves fine-grained distinctions and original context rather than functioning as a compressed summary alone.
E Limitations
The evaluation is limited to LoCoMo and PerLTQA, while xMemory adds construction overhead and depends on LLM-generated memory components whose extraction errors may affect retrieval.
- The empirical evaluation covers LoCoMo and PerLTQA but not all possible agent-memory settings.
- xMemory adds memory-construction cost through component extraction and maintenance of a revisable hierarchy.
- The implementation relies on LLM-generated memory components and summaries, so extraction errors may affect downstream retrieval.
F Broader Impact Statement
The broader-impact discussion describes possible benefits for long-term memory applications, highlights privacy and outdated-memory risks, and calls for safeguards in deployment. The appendix also documents dataset-specific answer-generation and memory-construction prompts.
- Broader impacts: xMemory may improve reliability and efficiency in personal assistants, long-term dialogue systems, and multi-session decision-support tools through lower token usage.
- Broader impacts: Potential deployment risks include privacy leakage, unintended retention of sensitive information, and reliance on outdated or incorrect memories.
- Broader impacts: Practical deployments should use consent, data minimisation, access control, retention and deletion policies, and mechanisms to inspect, correct, or override memories.
- Implementation details: The appendix includes prompts for boundary detection, episodic generation, component generation, group generation, and dataset-specific answer generation.
- Implementation details: The appendix uses boundary detection to segment dialogue, generates timestamped episodic records, and extracts persistent memory components from accumulated segments.
- Implementation details: Answer-generation prompts are tailored to LoCoMo and PerLTQA, including temporal resolution for LoCoMo and recency-based conflict handling for PerLTQA.