PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic-R: Learning to Retrieve for Agentic Search

Wenhan Liu, Xinyu Ma, Yutao Zhu, Yuchen Li, Daiting Shi, Dawei Yin, Zhicheng Dou

arXiv:2601.11888v1cs.IRcs.CL

TL;DR

Agentic search lacks retrievers tailored to its multi-turn, agent-generated queries, since similarity and single-turn utility signals do not fully capture passage usefulness. Agentic-R models utility through local relevance and global answer correctness, then iteratively co-optimizes the retriever and search agent; experiments across seven QA benchmarks report superior performance across search agents.

  • Problem

    Retriever design for agentic search is underexplored, while existing single-turn RAG utility methods and fixed-query training do not fit agent-generated multi-turn searches.

  • Method

    Agentic-R combines local query-passage relevance and global final-answer correctness with bidirectional iterative optimization of the search agent and retriever.

  • Results

    Agentic-R consistently outperforms strong baselines across different search agents on seven single-hop and multi-hop QA benchmarks.

  • Takeaways & Limitations

    The framework continuously improves retrieval using evolving, higher-quality agent-generated queries and can help agents solve questions with fewer search queries.

  • Takeaways & Limitations

    Evaluation uses single-hop and multi-hop QA benchmarks that do not fully represent deeper or more abstract reasoning settings such as expert-level or scientific reasoning tasks.

Abstract

from arXiv · show

Agentic search has recently emerged as a powerful paradigm, where an agent interleaves multi-step reasoning with on-demand retrieval to solve complex questions. Despite its success, how to design a retriever for agentic search remains largely underexplored. Existing search agents typically rely on similarity-based retrievers, while similar passages are not always useful for final answer generation. In this paper, we propose a novel retriever training framework tailored for agentic search. Unlike retrievers designed for single-turn retrieval-augmented generation (RAG) that only rely on local passage utility, we propose to use both local query-passage relevance and global answer correctness to measure passage utility in a multi-turn agentic search. We further introduce an iterative training strategy, where the search agent and the retriever are optimized bidirectionally and iteratively. Different from RAG retrievers that are only trained once with fixed questions, our retriever is continuously improved using evolving and higher-quality queries from the agent. Extensive experiments on seven single-hop and multi-hop QA benchmarks demonstrate that our retriever, termed \ours{}, consistently outperforms strong baselines across different search agents. Our codes are available at: https://github.com/8421BCD/Agentic-R.

1 Introduction

Agentic search extends RAG into multi-step reasoning and retrieval, but retriever design remains underexplored. Agentic-R addresses this gap by modeling passage utility from local relevance and global answer correctness, then iteratively optimizing the agent and retriever.

  • Motivation: Agentic search interleaves reasoning with retrieval, enabling agents to decompose complex questions into sequential sub-queries and gather evidence across turns.This extends traditional single-turn RAG's retrieval process into a multi-step search-during-think process.
  • Motivation: Existing agentic-search research emphasizes stronger agents while often relying on off-the-shelf similarity-based retrievers, even though semantic similarity does not guarantee answer usefulness.Retriever quality affects both agent training and final inference performance.
  • Research gap: Single-turn RAG utility methods do not directly transfer to agentic search because intermediate queries are agent-generated and lack the fixed-question, gold-answer setting.Existing retriever training also commonly uses one-way optimization with fixed training queries and a fixed generator.
  • Results: Experiments across seven single-hop and multi-hop QA benchmarks show consistent outperformance over strong baselines across different search agents.Further analysis reports that Agentic-R enables agents to solve questions with fewer search queries.
  • Approach: Agentic-R measures passage utility using both relevance to the current search query and contribution to final-answer correctness.The framework is designed specifically for multi-turn agentic search rather than single-turn RAG.
  • Approach: The framework iteratively optimizes the search agent and retriever bidirectionally, allowing them to evolve together through progressively improved agent-generated queries.This differs from retrievers trained once on fixed questions.

2 Related Work

Related work has improved search agents and trained retrievers for generation, but existing utility modeling and one-way training remain tailored to single-turn RAG. Agentic-R combines passage-utility modeling with iterative agent–retriever optimization for agentic search.

  • Agentic Search: Agentic-search research increasingly uses reinforcement learning to train agents that decompose questions, formulate sequential queries, and retrieve information across turns.Despite progress in agent optimization, the retriever remains a critical component influencing agent performance.
  • Training Retrievers for Generation: Utility-oriented retriever research addresses the gap between topical similarity and generation needs using generator feedback such as likelihoods, task metrics, or LLM annotations.These methods aim to align retrieval with downstream generation.
  • Training Retrievers for Generation: Existing passage-utility modeling and retriever training are limited to single-turn RAG and typically follow a one-way optimization paradigm.The paper positions agent–retriever iterative optimization as a framework tailored to agentic search.

3 Preliminary: Agentic Search

Agentic search alternates reasoning and external retrieval over multiple turns. At each turn, the agent analyzes missing information, issues a query, receives passages, and repeats the cycle until it produces a final answer.

  • Multi-turn search cycle: At each turn, the agent produces a reasoning trace, generates a search query, and receives passages from the retriever for subsequent reasoning.The retrieved set is incorporated into the agent's context.
  • Multi-turn search cycle: The reasoning–retrieval cycle repeats across iterations so the agent can refine its understanding and progressively gather relevant information.The agent terminates retrieval when it determines that the accumulated information is sufficient.
  • Process representation: The full process explicitly encloses reasoning, search queries, retrieved passages, and final answers within dedicated tags.The tags include <think>, <search>, <information>, and <answer>.

4 Methodology

Agentic-R constructs retriever-training data by measuring each candidate passage's local relevance and its effect on final-answer correctness, then trains the retriever with contrastive learning. It iteratively optimizes the search agent and retriever so improved agents generate new training queries.

  • Training Data Construction: For each intermediate query, Agentic-R retrieves 20 candidate passages and evaluates each using local relevance and global answer correctness.These signals distinguish positive and negative passages for retriever training.
  • Local Relevance: Local relevance is assigned by an LLM-based listwise scorer that compares candidate passages and gives each a score from 0 to 100.The scorer can incorporate an inferred sub-answer for the intermediate query when one is reliable.
  • Final Answer Correctness: Global answer correctness measures whether continuing the agent from a candidate passage produces the correct final answer under the preceding trajectory.The generated answer is compared with the gold answer using exact match.
  • Positive and Negative Passages Selection: Candidates are ranked first by global answer correctness and then by local relevance, with the top passage selected positive and lower-ranked passages sampled as negatives.A positive requires exact-match correctness and local relevance of at least 60; each training instance contains 16 passages.
  • Training Approach: The retriever encodes the intermediate query together with the original question and learns with contrastive loss using sampled, in-batch, and cross-device negatives.Previous-turn queries are excluded because the authors report that they introduce retrieval noise and degrade final performance.
  • Agent-Retriever Iterative Optimization: Agentic-R alternates search-agent and retriever optimization, allowing improved retrieval to support agent reinforcement learning and improved agents to produce new training queries.The search agent is trained with PPO and exact-match final-answer rewards.

5 Experiment

Experiments across seven QA benchmarks and three search agents show that Agentic-R improves average answer accuracy, especially for multi-hop QA, while iterative optimization and combined utility signals contribute to gains and fewer search turns.

  • 5.2 Overall Performance: Agentic-R achieves the best average EM across all three search agents, outperforming the second-best baseline by about 3.2 points in-domain and roughly 2 points out-of-domain.The result indicates generalization beyond the search agent used during training.
  • 5.2 Overall Performance: Agentic-R improves multi-hop QA more than single-hop QA, with an average gap of about 3 points over REPLUG versus 2 points on single-hop datasets.This comparison is reported using the authors’ search agent.
  • 5.2 Overall Performance: RAG-specific retrievers do not consistently outperform general-purpose retrievers in agentic search; LLM-Embedder and SCARLet are often inferior to E5.The authors attribute this pattern to mismatched utility signals and a distribution gap between user questions and agent-generated queries.
  • 5.3 Ablation Study: Replacing Agentic-R2 with Agentic-R1 causes an average drop of about 0.9 points, while replacing Agent2 with Agent1 causes an additional drop of about 1.9 points.These ablations support benefits from later retriever and agent iterations.
  • 5.3 Ablation Study: Removing global answer correctness or local relevance causes average drops of about 1.1 and 1.7 points, respectively, confirming that both utility signals support retriever training.The larger drop without local relevance indicates a stronger measured sensitivity to that component in this ablation.
  • 5.4 Search Turns Analysis: Agentic-R reduces average search turns by approximately 10% on HotpotQA and 15% on TriviaQA compared with REPLUG.The comparison uses the authors’ search agent and an Agentic-R retriever trained after two iterations.
  • 5.5 Different Iteration Number K: Performance improves through the first two optimization iterations but slightly degrades at the third, indicating convergence after two iterations.Agent1 + Agentic-R1 exceeds Agent1 + E5 by approximately 2.6 points, and Agent2 + Agentic-R1 improves over Agent1 + Agentic-R1 by about 1.9 points.

6 Conclusion

The paper presents Agentic-R, a retriever framework for agentic search that combines local relevance with global answer correctness and iteratively trains the retriever with the search agent. Experiments show superior retriever performance and fewer search turns.

  • Agentic-R models passage utility using both local relevance and global answer correctness for multi-turn agentic search.The framework uses these signals as supervision for retriever training.
  • The iterative agent–retriever framework continuously improves the retriever using higher-quality agent-generated queries.Training alternates between the search agent and retriever.
  • Agentic-R achieves superior performance on multi-hop and single-hop QA benchmarks and reduces the number of search turns required by the agent.The conclusion links the efficiency result to agentic search performance.

Limitations

The paper evaluates Agentic-R on standard single-hop and multi-hop QA benchmarks and uses moderately sized agents and retriever backbones. Its baseline comparisons include general-purpose embedding models and RAG-specific retrievers.

  • Scope of Evaluation: Evaluation covers widely used single-hop and multi-hop question answering benchmarks, which do not fully represent expert-level or scientific reasoning tasks.The authors specifically identify GPQA-like settings as a broader challenge for future evaluation.
  • Model Scale: Computational and memory constraints limit experiments to moderately sized search agents and retriever backbones.The authors report favorable scaling trends but leave validation with larger models for future work.
  • Compared Methods: Baseline retrievers comprise general-purpose embedding models and RAG-specific retrievers.E5 and BGE represent general-purpose encoders, while LLM-Embedder, SCARLet, and REPLUG model utility for RAG.
  • Compared Methods: REPLUG is reimplemented and trained on HotpotQA and TriviaQA with Qwen2.5-7B-Base because its official checkpoint is unavailable.This setup is used to ensure a fair comparison.

B Implementation Details

The implementation initializes the retriever from E5-base-v2, trains it contrastively with agent-generated supervision, and alternates retriever and search-agent optimization. The search agent uses PPO with masking that excludes retrieved tokens from policy updates.

  • Retriever Training: The retriever starts from e5-base-v2 and uses contrastive learning with one positive and multiple negative passages per query.Each query has N = 16 training passages in total.
  • Retriever Training: Retriever training uses mean pooling, ℓ2 normalization, two epochs, and in-batch and cross-device negatives.The stated learning rate is 2 × 10^-5, with temperature 0.01 and maximum input length 512 tokens.
  • Agent Training: The search agent is optimized with PPO, while token-level loss masking restricts policy updates to LLM-generated tokens.Retrieved tokens are excluded from gradient updates to stabilize search-augmented generation training.
  • Iterative Training: Agent–retriever optimization alternates between fixing the retriever while training the agent and using the trained agent to generate trajectories for retriever training.The setup performs two iterations in total.
  • Infrastructure: The implementation uses single-node training with 8 A800 80G GPUs and applies gradient checkpointing and FSDP with CPU offloading.These choices reduce memory consumption during training.

C.1 Different Backbone of Agentic-R

Agentic-R is tested with BGE-base and E5-large backbones in addition to E5-base. It outperforms baseline retrievers across tested backbones and search agents, with larger retriever capacity producing stronger results.

  • Cross-Backbone Performance: Agentic-R consistently outperforms all baseline retrievers across the tested backbones and search agents.The search agent and Agentic-R are trained for one iteration in this experiment.
  • Cross-Backbone Performance: Approximately 2.8 average EM points separate Agentic-RBGE-base from BGE-base under the authors’ search agent.This comparison measures improvement over the corresponding backbone retriever.
  • Retriever Capacity: E5-large consistently outperforms E5-base, and Agentic-RE5-large similarly outperforms Agentic-RE5-base.The results show a scaling trend with respect to retriever capacity.

C.2 Training Input

The default retriever input uses only the original question and current-turn query rather than historical queries. Adding historical queries reduces average performance, especially for the out-of-domain R1-Searcher.

  • Training Input: The default retriever input contains the original question Q and current-turn query q_i, excluding previous-turn queries.The ablation compares this default format with concatenated historical queries.
  • Training Input: Historical-query input concatenates Q, q_1 through q_{i−1}, and q_i with separator tokens.The same format is used during inference.
  • Ablation Results: Adding historical queries degrades Agentic-R’s average performance under both the in-domain search agent and out-of-domain R1-Searcher.Under R1-Searcher, the variant even underperforms general-purpose retrievers such as E5 and REPLUG.
  • Ablation Results: The authors attribute this degradation to extra noise because current agentic-search queries usually express clear intent without anaphoric references or contextual ambiguity.Historical queries therefore provide limited additional information while distracting the retriever from the current query’s core intent.
  • Case Study: A case study compares E5 and Agentic-R on a multi-hop HotpotQA query, distinguishing irrelevant information from key evidence in the retrieved trajectories.The comparison examines how retrieved passages affect generation of the correct answer.
Loading 2601.11888v1…