Source-linked AI summary
SE-Search: Self-Evolving Search Agent via Memory and Dense Reward
Jian Li, Yizhang Jin, Dongqi Liu, Hang Ding, Jiafu Wu, Dongsheng Chen, Yunhang Shen, Yulei Qin, Ying Tai, Chengjie Wang, Xiaotong Yuan, Yabiao Wang
TL;DR
Existing RAG and search-agent methods struggle with noisy retrieval and sparse feedback during autonomous, multi-turn information seeking. SE-Search addresses these issues with memory purification, atomic queries, and dense rewards, and consistently outperforms existing methods across single-hop and multi-hop QA benchmarks. The method remains limited by static retrieval, underexplored highly complex tasks, manually tuned reward hyperparameters, and support for only one tool.
Problem
Existing search agents can accumulate irrelevant or noisy documents and rely on limited feedback while autonomously searching for information.
Method
SE-Search combines memory purification, atomic query generation, and dense Query, Memory, Outcome, and Format rewards to improve autonomous search behavior.
Results
SE-Search consistently outperforms existing methods across seven single-hop and multi-hop QA benchmarks, achieving an average EM accuracy of 0.420.
Takeaways & Limitations
SE-Search enhances LLM self-memory, self-exploration, and self-discipline during information seeking.
Takeaways & Limitations
SE-Search uses a static retrieval corpus, underexplores highly complex tasks, requires manual dense-reward tuning, and supports only a single tool.
Abstract
from arXiv · showhide
Retrieval augmented generation (RAG) reduces hallucinations and factual errors in large language models (LLMs) by conditioning generation on retrieved external knowledge. Recent search agents further cast RAG as an autonomous, multi-turn information-seeking process. However, existing methods often accumulate irrelevant or noisy documents and rely on sparse reinforcement learning signals. We propose \textbf{S}elf-\textbf{E}volving \textbf{Search}, a Self-Evolving Search agent that improves online search behavior through three components, memory purification, atomic query training, and dense rewards. SE-Search follows a \textit{Think-Search-Memorize} strategy that retains salient evidence while filtering irrelevant content. Atomic query training promotes shorter and more diverse queries, improving evidence acquisition. Dense rewards provide fine-grained feedback that speeds training. Experiments on single-hop and multi-hop question answering benchmarks show that \texttt{SE-Search-3B} outperforms strong baselines, yielding a $10.8$ point absolute improvement and a $33.8\%$ relative gain over Search-R1.\footnote{We will make the code and model weights publicly available upon acceptance.}
1 Introduction
SE-Search targets the limitations of fixed RAG and existing search agents by filtering noisy evidence, encouraging diverse search behavior, and providing denser training feedback. It introduces a self-evolving search agent built around memory purification, atomic queries, and dense rewards.
- Fixed RAG pipelines limit an LLM’s ability to decide when and what to search for, reducing flexibility in real-world applications.
- Existing search agents often retrieve irrelevant or noisy documents and generate similar queries across search steps, limiting useful evidence acquisition.
- SE-Search improves autonomous search behavior through Memory Purification, Atomic Query, and Dense Rewards.
- Memory Purification extracts useful evidence into self-memory, Atomic Query promotes distinct multi-step searches, and Dense Rewards provide fine-grained feedback through Query, Memory, Outcome, and Format components.
- SE-Search is evaluated on seven diverse and challenging question-answering benchmarks.
2 Problem Formulation
The paper formulates question answering as generation conditioned on retrieved knowledge, then extends fixed retrieval with iterative reasoning and adaptive search. Subqueries retrieve intermediate knowledge that is integrated into the reasoning trajectory before producing the answer.
- The question-answering formulation represents the user question as Q, the answer as a, and the language model as πθ.
- RAG retrieves knowledge k from an external corpus D using retriever R(·), then generates an answer conditioned on Q and k.
- Fixed retrieval is insufficient for complex knowledge discovery tasks requiring iterative reasoning and adaptive retrieval.
- Search agents interleave retrieval with reasoning, allowing model output to include intermediate reasoning steps r and a final answer a.
- Subqueries qt retrieve intermediate knowledge kt, which is integrated into the reasoning trajectory through the operator ⊗.
3 Approach
SE-Search trains an autonomous search agent to gather evidence through interleaved reasoning, searching, memory updates, and answering. Its approach combines memory purification, atomic query rewards, dense trajectory feedback, and GRPO-based reinforcement learning.
- 3.1 Self-Evolving Search Agent: SE-Search represents search as trajectories that interleave thinking, queries, retrieved documents, distilled memory, and final answers.The agent selects actions from <think>, <search>, <documents>, <memory>, and <answer> while gathering evidence across steps.
- 3.1 Self-Evolving Search Agent: The optimization objective accounts for search queries, memory, and final answers, then uses rule-based rewards because ground-truth trajectories are unavailable.Retrieved tokens are approximated through search queries and memory contents, and reinforcement learning optimizes trajectory generation.
- 3.2 Memory Purification: Memory purification filters and consolidates relevant facts from retrieved documents into evolving memory, using a memory reward based on Cover Exact Match.The prompt supplements or revises previous memory with retrieved knowledge, while selective extraction reduces irrelevant content.
- 3.3 Atomic Query: Atomic query counting constrains query length and enforces diversity, with query rewards guiding when and how the agent invokes search.The reward encourages diverse queries and discourages unnecessary search after correct answers while encouraging more search after incorrect answers.
- 3.4 Dense Rewards: Dense rewards combine answer correctness, memory quality, query behavior, and format adherence, with weighted components and a decaying query-reward influence.Format penalties address excessive search turns, invalid actions, and unmatched or incorrectly ordered special tokens; the query reward is reduced through cosine decay.
- 3.4 Dense Rewards: SE-Search uses GRPO to optimize the agent without a separate value estimator, estimating advantages from normalized rewards across grouped sampled outputs.External retrieval tokens are masked during loss computation, and the overall dense reward supplies the reward signal.
4 Experiments
SE-Search is evaluated across single-hop and multi-hop QA benchmarks against retrieval-free, retrieval-based, and search-agent baselines. It improves accuracy through its components and shows adaptive search behavior, stronger multi-hop gains, and scaling with backbone size.
- Main Performance: SE-Search achieves an average EM accuracy of 0.420 and consistently outperforms Search-R1, InForage, and AutoRefine across seven single-hop and multi-hop benchmarks.The comparison uses the same retriever, corpus, training data, and Qwen2.5-3B setting.
- Main Performance: 4.5 percentage points and 11.1% relative gain on HotpotQA, plus 8 percentage points and 23.2% relative gain on Bamboogle, are reported over AutoRefine.The gains are particularly evident on complex multi-hop QA tasks.
- Ablation Studies: Memory Purification improves Musique by 5 points and Bamboogle by 9.1 points, while Atomic Query adds relative gains of 15.87% on 2Wiki, 35.83% on Musique, and 42.69% on Bamboogle.The ablation compares controlled configurations built on Search-R1 and evaluates each component across seven benchmarks.
- Generalization and Scalability: SE-Search’s average EM and F1 increase across Qwen2.5-3B, 7B, and 14B backbones, with improvements on most datasets as model scale increases.The retriever, corpus, and evaluation protocol remain unchanged while only backbone scale varies.
- Additional Analysis: Accuracy rises from 0.36 to approximately 0.41 while average search calls fall from 1.53 to 1.32, both changing by about 14%.Memory Purification recalls more documents early, whereas Atomic Query produces shorter, diverse queries that retrieve fewer, more relevant documents.
- Additional Analysis: SE-Search averages approximately 1.54 search calls on multi-hop questions while remaining close to 1.0 on single-hop benchmarks.This contrasts with Search-R1’s fixed single search call and indicates search frequency varies with question type.
- Additional Analysis: SE-Search uses shorter queries at approximately 50 characters than AutoRefine at approximately 65 and Search-R1 at approximately 90, while memory CEM reaches approximately 0.65.The reported statistics connect atomic querying with query focus and Memory Purification with answer-relevant stored evidence.
- Additional Analysis: Dense rewards improve early memory-content accuracy and accelerate convergence by discouraging invalid outputs and providing graded F1-based feedback.The format reward reduces unproductive exploration trajectories, while the F1-based outcome reward supplies denser feedback than EM.
5 Related Works
Related work includes conventional RAG, autonomous search agents, and reinforcement-learning methods for improving search behavior. SE-Search addresses accumulated retrieval noise by combining Think-Search-Memorize with Atomic Query.
- Retrieval-Augmented Generation: Conventional RAG commonly uses a fixed retrieve-then-generate workflow, limiting the model’s ability to decide when to search during reasoning.Search agents instead invoke search tools within multi-step reasoning processes.
- Search Agents: Search-agent methods combine LLM intent interpretation and planning with multi-turn search actions that accumulate retrieved evidence.Representative systems include Search-o1, WebThinker, Web-Dancer, and Search-R1.
- Search Behavior Training: A common limitation is that retrieved context accumulates noise across steps and weakens later reasoning; SE-Search responds with Think-Search-Memorize and Atomic Query.The related methods include Search-R1, ReSearch, AutoRefine, and WebSailor.
- Reinforcement Learning: Reinforcement-learning approaches in this area include PPO, DPO, GRPO, DAPO, and GSPO, which target LLM behavior and training stability or efficiency.These methods provide the broader optimization context for search-agent training.
6 Conclusions
SE-Search enhances LLM search behavior through memory purification, atomic queries, and dense rewards, addressing noisy retrieval and limited feedback in prior methods. Experiments show consistent gains on single-hop and multi-hop question answering benchmarks.
- SE-Search enhances LLM self-memory, self-exploration, and self-discipline through memory purification, atomic query, and dense reward mechanisms.
7 Limitations
SE-Search is evaluated with a static retrieval corpus rather than live web search and does not cover all complex search tasks or tool capabilities. Its dense-reward design also requires manual hyperparameter selection and tuning.
- SE-Search does not use live web search because its retrieval corpus is static and lacks real-time internet information.
- Highly complex tasks such as BrowseComp remain underexplored.
- Dense rewards require manual selection and tuning of several hyperparameters.
- SE-Search supports only a single tool and lacks capabilities such as page browsing and code execution.
A.1 Training and Evaluation Datasets
The training dataset merges NQ and HotpotQA, while evaluation spans seven benchmarks using both test and development sets. The resulting training and evaluation collections contain 169,615 and 51,713 samples, respectively.
- 169,615 samples form the training dataset by merging NQ and HotpotQA.
- 51,713 samples comprise evaluation data from seven benchmarks, using four test sets and three development sets.The test-set benchmarks are NQ, TriviaQA, PopQA, and Bamboogle; the development-set benchmarks are HotpotQA, 2Wiki, and Musique.
A.2 GRPO Hyperparameters
The GRPO setup follows Search-R1 and AutoRefine for core data, actor, and rollout configurations, while dense-reward hyperparameters specify query-length bounds, a similarity threshold, and a decay step.
- The VeRL configuration uses the same data, actor, and rollout setup as Search-R1 and AutoRefine.
- Atomic query rewards apply to query lengths from 10 to 120 with a similarity threshold of 0.3.
- The dense-reward decay step is 150, equal to half of the total training steps.
A.3 More Experiments Results
SE-Search is characterized through training dynamics across seven benchmarks and case studies of atomic queries, memory contents, and full search trajectories. These analyses illustrate the agent’s self-planning, self-exploration, self-memory, and self-discipline.
- Training Dynamics: Training dynamics are evaluated as average accuracy across seven benchmarks: NQ, TriviaQA, PopQA, Bamboogle, HotpotQA, 2Wiki, and Musique.Figure 5 presents SE-Search’s evolution of EM accuracy across the same seven benchmarks.
- Atomic Queries: Atomic queries derived from a complex user question illustrate SE-Search’s self-planning and self-exploration.Table 5 lists the atomic search queries used for this analysis.
- Case Studies: Memory contents and full trajectories from Musique and HotpotQA illustrate SE-Search’s self-memory and self-discipline.Tables 6 and 7 present two randomly selected examples, with key evidence highlighted in blue.