PaperlayerView exact revision on arXiv

Source-linked AI summary

MemRetriever: Learning to Search, Reflect, and Retrieve from Long-Term Memory

Ruiyang Jiang, Chunyu Li, Zhiyu Li

arXiv:2609.11951v1cs.IR

TL;DR

Static top-k memory retrieval can miss distributed evidence and retain irrelevant content, motivating a more adaptive retrieval process. MemRetriever uses a multi-step search agent with search, denoising, and termination decisions, and it improves memory retrieval and multi-hop question answering across the evaluated benchmarks. Its storage-agnostic decision logic also supports broader retrieval-augmented generation settings, while similarity-based reward designs can retain high-similarity but low-value evidence.

  • Problem

    Static top-k retrieval may miss evidence distributed across conversations and retain insufficiently relevant content for complex questions.

  • Method

    MemRetriever trains a search agent to iteratively choose parallel search, serial search, or reflection-based denoising, using rewards for coverage, evidence preservation, and answer sufficiency.

  • Results

    MemRetriever improves long-term memory retrieval and complex multi-hop question answering across LOCOMO, LongMemEval, HotpotQA, MuSiQue, and 2WikiMultiHopQA.

  • Takeaways & Limitations

    An intermediate retrieval decision layer can actively control searching, evidence filtering, and stopping, with logic that is independent of the storage backend.

  • Takeaways & Limitations

    Similarity-based reflection rewards can favor semantically similar evidence that lacks key facts, causing evidence accumulation without corresponding answer-quality gains.

Abstract

from arXiv · show

Long-term memory enables personalized agents, but its value depends on retrieving the right evidence at the right time. Most memory systems use static top-k retrieval: they issue one query, return a fixed number of memories, and pass them directly to a downstream model. This approach can miss evidence distributed across sessions, introduce irrelevant content, and waste context, especially for multi-hop, temporal, and knowledge-update questions. We present MemRetriever, an agentic retrieval model that treats memory access as a multi-step search process. At each step, MemRetriever reasons over the current evidence and selects parallel search for broad exploration, serial search for targeted completion, or reflection and denoising for filtering and evidence assessment. It stops when the retained evidence is sufficient for downstream answering. We construct ReAct-style search-memory trajectories for supervised warm-start training and further optimize the model with Group Relative Policy Optimization. The reward design encourages evidence coverage, noise reduction, answer sufficiency, and efficient termination. Experiments on LOCOMO, LongMemEval, HotpotQA, MuSiQue, and 2WikiMultiHopQA show consistent improvements over static retrieval and supervised-only baselines. MemRetriever-4B-RL also outperforms DeepSeek-v4-Flash on the main LongMemEval retrieval metrics under the same pipeline and achieves the strongest results among the compared methods on MuSiQue. Because its decision logic is independent of the storage backend, MemRetriever can also operate over external knowledge bases and vector databases. These results show that an intermediate decision layer that plans, searches, filters evidence, and determines when to stop can improve both long-term memory retrieval and knowledge-intensive question answering.

1 Introduction

MemRetriever reframes long-term memory access as multi-step agentic search that plans retrieval, filters evidence, and stops when support is sufficient. Its coverage- and sufficiency-aligned rewards improve retrieval over static top-k baselines across memory and QA datasets.

  • 1 Introduction: Static top-k retrieval can miss distributed multi-hop evidence and cannot adapt the number of retrieved memories to each question’s evidence needs.A single query may retrieve only part of the required evidence, while fixed k can be insufficient for complex questions or excessive for simple ones.
  • 1 Introduction: The search agent reasons over current evidence, chooses broad parallel or targeted serial search, and reflects to remove noise before deciding whether evidence is sufficient.These actions support exploration, completion of identified gaps, evidence filtering, and adaptive termination.
  • 1 Introduction: The retrieval decision layer is storage-agnostic and can operate between a model and either a memory store or an external vector knowledge base.The paper illustrates this broader deployment scope with vector databases such as Milvus.
  • 1 Introduction: MemRetriever formulates memory retrieval as a multi-step agentic search process with parallel search, serial search, and reflection-based denoising.The search agent constructs an evidence pool rather than directly answering the question.
  • 1 Introduction: A hierarchical reward function encourages incremental evidence coverage, recall-preserving denoising, and answer sufficiency at termination.The search, reflection, and termination rewards are aligned with their corresponding stages of the multi-step process.
  • 1 Introduction: Experiments on HotpotQA and memory datasets such as LOCOMO show significant multi-metric improvements over static top-k retrieval baselines.The method is evaluated for both complex question answering and long-term memory retrieval.

2 Related Work

Related work spans static RAG, reinforcement-learned agentic search, long-term memory systems, and tool-augmented agents. MemRetriever connects these lines by making memory access an explicit multi-step process that searches, purifies evidence, and interfaces with varied storage backends.

  • 2.1 Agentic Search: RAG and agentic-search research address external knowledge acquisition, while memory-system research supports persistent personalized interaction but commonly relies on similarity retrieval.As memory scales, simple top-k retrieval can introduce noise and efficiency degradation.
  • 2.2 Memory Retrieval in Agent Memory: Prior memory retrieval methods mainly use indexing, ranking, or single-stage filtering and do not explicitly decide whether evidence is missing, sufficient, or removable.This limits their ability to manage evidence through an explicit multi-step search process.
  • 2.4 Our Contributions: Unlike general agentic search work focused on open-domain or external-document retrieval, this paper applies agentic search to long-term agent memory.MemSearcher also connects these areas but primarily addresses interaction-history expansion, whereas this paper targets active evidence search and completion before answering.
  • 2.4 Our Contributions: MemRetriever advances memory retrieval from static top-k selection to dynamic evidence discovery and purification through parallel search, serial search, and reflection-based denoising.Parallel search explores ambiguous directions, serial search fills identified gaps, and reflection filters noisy or redundant results.
  • 2.4 Our Contributions: The search agent functions as an independent intermediate decision layer that can plan retrieval across memory stores or external vector knowledge bases.Its modeled decisions concern when to retrieve, how to retrieve, and when to stop, enabling storage-agnostic deployment.

3 Method

MemRetriever formulates memory retrieval as a multi-step ReAct-style process that builds a compact evidence pool through adaptive search, reflection, and stopping decisions. Its training combines supervised trajectory warm-starting with rewards for coverage, denoising, answer sufficiency, and efficient retrieval.

  • 3 Method: MemRetriever constructs a compact evidence pool through iterative ReAct-style search rather than one-shot top-k selection.The agent observes the question, current evidence, and search history while updating the pool through repeated actions.
  • 3 Method: The action space combines parallel search for ambiguous evidence, serial search for targeted gaps, and reflection-and-denoise for filtering and stopping.Reflection can set done to true, terminating the loop and returning the final evidence pool.
  • 3 Method: The policy objective rewards learning when to retrieve, how to retrieve, how to denoise, and when to stop across complete retrieval trajectories.The total reward combines search, reflection, stop, and format components.
  • 3.1 Supervised Warm Start: Supervised warm-start trajectories teach reasoning, tool calls, structured arguments, and output formatting before reinforcement learning.This initialization reduces invalid JSON, incomplete tool calls, formatting errors, and incorrect search actions that would otherwise add noise to reward evaluation.
  • 3.2 Reward Design for Memory Retrieval: Search rewards use incremental gold-evidence coverage with query costs, while reflection filters redundancy and improves evidence density.Incremental coverage encourages new information and alleviates reward sparsity, whereas query penalties discourage unnecessary searches.
  • 3.2 Reward Design for Memory Retrieval: Similarity-based reflection rewards can retain semantically similar but factually inadequate memories, motivating coverage-based rewards and penalties for removing gold-evidence coverage.The revised design ties reward to key facts and addresses mistaken removal of important evidence during deduplication.

4 Model Architecture

MemRetriever uses a ReAct-style search agent to iteratively retrieve, assess, and denoise memory evidence before downstream answering. It selects among parallel search, serial search, and reflection-based denoising until the evidence pool is sufficient.

  • 4 Model Architecture: MemRetriever iteratively updates an evidence pool through reasoning, tool actions, observations, and reflection until it is sufficient for downstream answering.The model serves as an intermediate search agent rather than directly producing the user-facing answer.
  • 4 Model Architecture: The architecture explicitly addresses search necessity, strategy selection, evidence quality assessment, and stopping conditions.These decisions determine whether to continue searching, which action to invoke, and whether the evidence can support an answer.
  • 4 Model Architecture: Parallel search expands recall across complementary directions when evidence needs are ambiguous or multifaceted.It is used when the evidence space remains unclear and multiple retrieval paths may be relevant.
  • 4 Model Architecture: Serial search targets one concrete missing fact, relation, entity, time point, or attribute after the information gap has been localized.This action is intended for precise completion rather than broad exploration.
  • 4 Model Architecture: Reflection-based denoising filters irrelevant, redundant, noisy, or misleading evidence and can terminate retrieval when the retained pool is sufficient.The downstream reader receives a more compact, relevant, and reasoning-ready evidence pool.

5 Training

MemRetriever is trained in stages that construct ReAct search-memory trajectories, use supervised warm-start fine-tuning, and then optimize multi-round retrieval with GRPO. The training process addresses search decisions, trajectory quality, and reward-aligned evidence acquisition.

  • 5 Training: The training pipeline consists of ReAct trajectory construction, supervised warm start, and GRPO-based reinforcement learning.It teaches the model whether to search, what to search for, which tool to call, how to interpret observations, and when to stop.
  • 5 Training: Teacher-generated Think–Action–Observation trajectories train decisions about evidence gaps, tool selection, and answer sufficiency across retrieval rounds.GPT-4.1-mini generates trajectories while connected to a real vector retrieval backend.
  • 5 Training: A lightweight plan prompt constrains later retrieval and reduces search drift as the number of search rounds increases.The authors report that first-round reasoning is usually the most accurate and use it to guide subsequent retrieval.
  • 5 Training: Quality filtering removes invalid or low-quality trajectories using format validation, JSON parsing, tool-call consistency, and retrieval-logic checks.A trajectory is accepted only when the model judges the task answerable and recommends done next.
  • 5 Training: Supervised warm start fine-tunes Qwen3-4B-Thinking on constructed ReAct trajectories before GRPO performs multi-round retrieval optimization.GRPO is selected because search rewards are trajectory-level and relative in nature.
  • 5 Training: The reward design encourages new relevant information, preservation of answer-supporting evidence, removal of distracting content, and stopping when evidence is sufficient.These objectives align search, reflection, and termination with high-quality evidence acquisition.

6 Experiments

Across five benchmarks, MemRetriever improves long-term memory retrieval and multi-hop question answering over static and supervised-only approaches. Its strongest gains arise from reinforcement-learned evidence acquisition, reflection-based denoising, and answer-sufficiency-aware termination.

  • 6.2 LOCOMO Results: On LOCOMO, MemRetriever-4B-RL raises LLM-as-Judge accuracy from 0.785 to 0.830 over MemRetriever-4B-SFT while reducing average token consumption.All agentic variants use the same parallel-search, serial-search, and reflection-based denoising pipeline.
  • 6.3 LongMemEval Results: On LongMemEval, MemRetriever-4B-RL improves the SFT model across main retrieval metrics and answer quality while using fewer tokens on average.The results support learning when further retrieval is useful and when the evidence pool is sufficient.
  • 6.4 QA Results: MemRetriever-4B-RL achieves the best EM, F1, and judge score among listed methods on MuSiQue.The result is especially notable on a benchmark designed to distinguish genuine multi-step reasoning from shallow pattern matching.
  • 6.5 Analysis: MemRetriever brings stable improvements in retrieval and complex question answering across LOCOMO, LongMemEval, HotpotQA, MuSiQue, and 2WikiMultiHopQA.Benefits are particularly clear as reasoning chains lengthen and more evidence pieces are required.
  • 6.5 Analysis: Coverage-driven search, coverage-preserving reflection, and judge-based termination rewards guide evidence discovery, denoising, and answer sufficiency.Reflection reduces irrelevant evidence while preserving useful evidence, and termination directly optimizes final answer sufficiency.
  • 6.5 Analysis: MemRetriever remains beneficial across different retrieval backends and answer-model settings, supporting its use as an independent intermediate module.The method is compared with representative search-augmented baselines and is designed to improve evidence discovery and final answer quality.

7 Conclusion

MemRetriever reframes long-term memory access as an agentic, multi-step decision process that plans searches, filters evidence, and stops when the retained evidence is sufficient. Across memory and multi-hop QA benchmarks, it improves retrieval and answering while the RL-trained model also reduces unnecessary token consumption.

  • MemRetriever turns memory retrieval from static top-k matching into multi-step planning, search-direction control, evidence denoising, and sufficiency-based stopping.The model actively manages the evidence pool before downstream answering.
  • ReAct-style trajectories provide supervised warm-start training, followed by GRPO optimization with rewards for coverage, evidence preservation, noise reduction, and efficient termination.These components train the model to decide what to retrieve, how to refine evidence, and when to stop.
  • Across LOCOMO, LongMemEval, HotpotQA, MuSiQue, and 2WikiMultiHopQA, MemRetriever improves long-term memory retrieval and complex multi-hop question answering.The gains are especially clear when questions require multiple evidence pieces, longer reasoning chains, or dynamic filtering.
  • Compared with SFT, RL-trained MemRetriever achieves better retrieval coverage and answer quality while reducing unnecessary token consumption on memory benchmarks.The reported pattern indicates gains in both effectiveness and efficiency.
  • The storage-agnostic decision logic can extend beyond long-term memory to vector databases and external knowledge stores.The broader conclusion is that retrieval can benefit from an intermediate layer controlling search, filtering, and stopping.

A Appendix A. Data Construction Prompt

The appendix specifies a tri-tool ReAct-style search-agent prompt for constructing search-memory trajectories. It requires explicit reasoning before tool calls and assigns distinct roles to parallel search, serial search, and reflection-based denoising.

  • The trajectory-construction prompt defines a tri-tool interaction protocol and prevents the search agent from answering the question directly.The agent instead supports a downstream answer model through tool-mediated evidence search.
  • Each response must explain known evidence, remaining gaps, tool choice, and expected progress before issuing the tool call.The required thought-and-call structure makes the search rationale explicit at every step.
  • Parallel search explores complementary directions, serial search fills one focused evidence gap, and reflection-and-denoise scores, filters, and summarizes the current evidence pool.The prompt permits reflection when useful rather than after every search.
  • The agent should use as few tool calls as necessary and stop only when a downstream answer model can answer without guessing.Complex questions may require more steps, while simple questions may need only one or two.

B Appendix B. Case Study

The case study illustrates how MemRetriever acquires and cleans evidence through multi-step search in a multi-hop knowledge question. It identifies an intermediate railroad owner, searches for that owner’s headquarters, and retains the key evidence while discarding irrelevant results.

  • Case 1: Multi-hop knowledge retrieval QA: The first retrieval establishes CSX Transportation as the Russell Subdivision’s owner, leaving the headquarters location as the remaining information gap.This demonstrates targeted sequential completion of a multi-hop chain.
  • Case 1: Multi-hop knowledge retrieval QA: Reflection-and-denoise retains the two key passages supporting the full chain and discards unrelated retrieved facts about railroad classification and routes.The retained evidence links Russell Subdivision ownership to CSX Transportation headquarters in Jacksonville, Florida.
  • Case 1: Multi-hop knowledge retrieval QA: The multi-hop case uses two serial searches followed by reflection to connect the Russell Subdivision to CSX Transportation and then to Jacksonville, Florida.The first search identifies the railroad owner; the second targets the headquarters location.
  • The figure contrasts retained evidence shown in green with discarded evidence shown in struck-through gray.Its second case also illustrates parallel search followed by reflection for personal memory retrieval.
Loading 2609.11951v1…