Source-linked AI summary
Learning to Retrieve from Agent Trajectories
Yuqi Zhou, Sunhao Dai, Changle Qu, Liang Pang, Jun Xu, Ji-Rong Wen
TL;DR
Human-centric retrieval training mismatches the query and result-use patterns of multi-step search agents. The paper analyzes agent trajectories, proposes LRAT to derive weighted retrieval supervision, and reports consistent improvements across deep research benchmarks. It presents trajectories as a practical supervision source for agent-centric retrieval.
Problem
Human-centric retrievers do not match how agents issue queries and consume results during multi-step reasoning, motivating direct training from agent interaction data.
Method
LRAT mines retrieval supervision from agent trajectories using browsing, unbrowsed documents, and post-browse reasoning traces, then weights examples by relevance intensity.
Results
Experiments on in-domain and out-of-domain deep research benchmarks show consistent improvements in evidence retrieval and end-to-end agent performance across diverse agent architectures and scales.
Takeaways & Limitations
Agent trajectories are presented as a practical and scalable supervision source for training retrieval models aligned with agentic search.
Abstract
from arXiv · showhide
Information retrieval (IR) systems have traditionally been designed and trained for human users, with learning-to-rank methods relying heavily on large-scale human interaction logs such as clicks and dwell time. With the rapid emergence of large language model (LLM) powered search agents, however, retrieval is increasingly consumed by agents rather than human beings, and is embedded as a core component within multi-turn reasoning and action loops. In this setting, retrieval models trained under human-centric assumptions exhibit a fundamental mismatch with the way agents issue queries and consume results. In this work, we argue that retrieval models for agentic search should be trained directly from agent interaction data. We introduce learning to retrieve from agent trajectories as a new training paradigm, where supervision is derived from multi-step agent interactions. Through a systematic analysis of search agent trajectories, we identify key behavioral signals that reveal document utility, including browsing actions, unbrowsed rejections, and post-browse reasoning traces. Guided by these insights, we propose LRAT, a simple yet effective framework that mines high-quality retrieval supervision from agent trajectories and incorporates relevance intensity through weighted optimization. Extensive experiments on both in-domain and out-of-domain deep research benchmarks demonstrate that retrievers trained with LRAT consistently improve evidence recall, end-to-end task success, and execution efficiency across diverse agent architectures and scales. Our results highlight agent trajectories as a practical and scalable supervision source, pointing to a promising direction for retrieval in the era of agentic search.
1. Introduction
Agentic search exposes a mismatch between human-centric retrieval training and how agents issue queries and use results. The paper proposes learning directly from agent trajectories and introduces LRAT to mine and weight retrieval supervision from these interactions.
- Motivation: Human-centric retrievers trained from clicks and dwell time encode assumptions that break when agents issue queries for intermediate reasoning objectives.Agent queries support multi-step problem solving rather than immediate informational needs, producing different relevance patterns.
- Learning from trajectories: Agent trajectories provide multi-step supervision from intermediate queries, retrieved documents, and reasoning steps, aligning retrieval objectives with agent feedback.Because trajectories arise from agent invocations, they can support a sustainable retrieval data flywheel.
- Behavioral signals: Trajectory analysis identifies browsing as necessary for task success, unbrowsed documents as reliable negatives, and post-browse reasoning as a relevance-intensity signal.These behaviors distinguish useful documents from superficially browsed ones.
- LRAT: LRAT mines query-document supervision from search-browse transitions, refines positives with post-browse reasoning traces, and weights examples by reasoning length.The instantiation uses 26,482 trajectories and 91,713 training pairs without additional human annotation.
- Results: Experiments on in-domain and out-of-domain deep research benchmarks show consistent gains in evidence retrieval and end-to-end agent performance across agent architectures and scales.The authors also report support for a self-improving data flywheel.
2. Related Work
Prior retrieval work largely optimized ranking from human judgments and implicit user feedback. Agentic search instead motivates retrieval optimization around multi-step agent interactions.
- Agentic search: Agentic search advances multi-step reasoning and iterative search interactions, but its underlying retrieval models have largely remained static off-the-shelf tools.Examples include search APIs, BM25, and dense embedding models.
- Human feedback: Learning-to-rank research began with explicit human relevance judgments and later used implicit user feedback to address manual-annotation scalability.These methods optimize document ordering with pointwise, pairwise, or listwise objectives.
3. Preliminaries
Deep Research Agents iteratively reason, search, browse, and synthesize evidence through multi-turn trajectories. The paper formalizes retrieval training from these trajectories to produce rankings that support agent problem solving.
- Deep Research Agents: Deep Research Agents solve complex information-seeking tasks through iterative interactions with an external retrieval system.Their execution trajectories capture the repeated reasoning and action process.
- Trajectory structure: At each turn, the agent maintains a reasoning state, selects an action, observes retrieved information, and decides whether more information is needed.The trajectory is represented as alternating reasoning, actions, and observations.
- Search: Search generates an intermediate query targeting an information gap and returns ranked candidate documents, initially observed through snippets and summaries.The agent uses these coarse signals for initial assessment and subsequent decisions.
- Browse and answer: Browse selects a previously retrieved candidate and returns its full content, which the agent incorporates into later reasoning and actions.The cycle continues until the agent has sufficient information to answer.
- Task motivation: Existing agents commonly use human-trained off-the-shelf retrievers, while their abundant multi-turn trajectories remain largely untapped for retrieval training.This motivates training retrievers directly from agent trajectories.
- Task definition: Learning to Retrieve from Agent Trajectories trains ranked document lists from execution trajectories to support an agent’s multi-step reasoning and problem-solving process.Its supervision is derived directly from agent behaviors rather than traditional learning-to-rank assumptions.
4. Analysis of Agent Trajectories
The analysis examines how deep research agents interact with retrieval systems, using generated trajectories to identify behaviors associated with effective information acquisition. Browsing, document rejection, and post-browse reasoning provide progressively richer signals of document utility.
- The study analyzes deep research agent trajectories to identify retrieval-interaction patterns that differ from human search and inform agent-centric relevance modeling.
- Environment Setup: Trajectories are generated on InfoSeekQA with Wiki-25-Dump, multiple retrievers, top-10 search candidates, and 64-token snippets before full-document browsing.The setup uses 10K verified seed queries and retrievers spanning BM25 and three dense embedding models.
- Browsing Is a Necessary Signal for Successful Retrieval: Successful trajectories have higher Browse-to-Search ratios and transition from [Search] to [Browse] more often than unsuccessful trajectories.Unsuccessful runs more frequently remain in search-only loops without progressing to document consumption.
- Browsing Is a Necessary Signal for Successful Retrieval: Task success increases monotonically with the number of browsed evidence documents and falls to zero when no required-evidence document is browsed.These observations motivate using browsed documents as primary positive-supervision candidates.
- Unbrowsed Documents as Negative Signals: Agent browsing is distributed relatively evenly across ranking positions, supporting unbrowsed documents as rejection signals rather than exposure-limited negatives.This contrasts with human click behavior concentrated at top ranks.
- Post-Browse Reasoning Traces Are Important Indicators: Longer post-browse reasoning is associated with correct trajectories and provides a finer-grained relevance signal than binary browsing feedback.Short reasoning often accompanies quickly abandoned documents judged unhelpful.
5. Learning to Retrieve from Trajectories
LRAT trains retrievers directly from deep-research agent trajectories by progressively mining relevance signals and weighting examples by inferred document utility. It combines browsing transitions, unbrowsed candidates, post-browse reasoning, and weighted contrastive learning to optimize retrieval for agent interactions.
- 5. Learning to Retrieve from Trajectories: LRAT progressively mines query-document supervision from agent trajectories, beginning with browsing decisions and refining positives using post-browse reasoning traces.The framework is designed to reflect how agents consume and utilize retrieval results.
- 5.1.1. Naive Relevance Mining from Search-Browse Transitions: A document browsed after query qt is treated as a naive positive because browsing is a necessary prerequisite for successful task completion.The resulting coarse instance pairs the query and browsed document with the other retrieved candidates as negatives.
- 5.1.1. Naive Relevance Mining from Search-Browse Transitions: Unbrowsed candidates in the same retrieved set are treated as naive negatives because agent browsing decisions show weak position dependence.This contrasts with human click logs, which require careful debiasing for position bias.
- 5.1.2. Reasoning-Aware Positive Refinement: An LLM verifier uses immediate post-browse reasoning to remove browsed-but-unhelpful documents while retaining 97.2% of ground-truth evidence documents.On BrowseComp-Plus annotations, it retains 74.8% of browsed non-evidence documents, indicating selective noise removal alongside agent-specific utility.
- 5.2. Relevance Intensity and Retriever Optimization: LRAT converts post-browse reasoning length into a bounded, normalized relevance-intensity weight and uses sample-wise weighting in a modified InfoNCE objective.The normalization sets E[w] ≈ 1, while deeper reasoning assigns greater training importance to the associated document.
- 5.2.2. Weighted Contrastive Learning: The hybrid negative set combines unbrowsed same-query candidates with in-batch negatives to separate high-utility evidence from rejected and unrelated documents.LRAT is instantiated with a standard bi-encoder dense retriever and evaluated on in-domain InfoSeek-Eval and out-of-domain BrowseComp-Plus benchmarks.
6. Experiments
The experiments evaluate LRAT across in-domain and out-of-domain benchmarks, diverse dense retrievers, and multiple search-agent architectures using task success, efficiency, and retrieval-quality metrics.
- Evaluation Setup: LRAT is evaluated on InfoSeek-Eval and BrowseComp-Plus to assess in-domain and out-of-domain generalization.InfoSeek-Eval contains 300 disjoint multi-hop queries, while BrowseComp-Plus contains 830 complex human-authored deep-research questions.
- Evaluation Setup: The study tests Multilingual-E5-Large-Instruct and Qwen3-Embedding-0.6B, representing encoder-based and decoder-based dense retrievers.
- Evaluation Setup: LRAT is integrated into six open-source search agents spanning task-optimized systems and generalist agentic foundation models.The agents range from AgentCPM-Explore-4B to GPT-OSS-120B, MiniMax-M2.1-229B, and GLM-4.7-358B.
- Evaluation Setup: Evaluation measures Success Rate, Average Step Count, and retrieval quality in realistic agentic settings.Success Rate is assessed by an automated LLM judge for answer correctness, while fewer average steps indicate more direct information acquisition.
- Implementation: Retriever training uses FlagEmbedding with 2 epochs, batch size 32, learning rate 1e−6, group size 10, and temperature 0.02.
6.2. Overall Performance
Across in-domain and out-of-domain evaluations, LRAT improves task success, evidence retrieval, and execution efficiency across retriever and agent backbones.
- Overall Results: LRAT consistently improves task success, evidence retrieval, and agent execution efficiency across retriever and agent backbones.
- Evidence Retrieval: Evidence recall on BrowseComp-Plus increases by 7% to over 37% across agents for both Qwen3-Embed and E5-Large.
- Task Success: LRAT-trained retrievers achieve higher success rates than baselines across in-domain and out-of-domain settings, agent architectures, and parameter scales.
- Task Success: Retrieval quality remains a critical bottleneck despite strong agent capabilities.
- Execution Efficiency: Average interaction steps decrease by up to ∼30% on InfoSeek-Eval with LRAT-trained retrievers.The reported reduction indicates more precise evidence acquisition with fewer exploratory interactions.
6.3. Ablation Study
The ablation study shows that LRAT gains accumulate as trajectory supervision progresses from browsing signals to filtered and reasoning-aware relevance signals.
- Naive Supervision: Using browsed documents as positives and all other documents as negatives yields substantial performance gains.This variant is called +Naive.
- Filtering: LLM-based filtering removes browsed documents that do not meaningfully contribute to subsequent reasoning, improving supervision quality.
- Reweighting: Reasoning-length reweighting improves performance by modeling heterogeneous document contributions through relevance intensity.
- Overall Ablation: Trajectory-derived supervision becomes increasingly effective as richer agent signals are incorporated into retriever training.
6.4. Scalability and Robustness Analysis
LRAT benefits from additional trajectory data and remains stronger than the base retriever across varying retrieval budgets, although excessively large top-K values can hurt performance.
- Inference-time Robustness: LRAT is evaluated under varying training-data sizes and retrieval top-K settings.
- Trajectory Data: The trajectory correctness ablation uses 10K correct and 10K incorrect examples.
- Training-time Scalability: Agent success rates generally improve as the LRAT training dataset grows across evaluated models.The scalability analysis uses 30K InfoSeekQA trajectories collected with Tongyi-DeepResearch and BM25.
- Inference-time Robustness: Increasing top-K does not always improve agent performance monotonically, because overly large retrieval budgets can increase noise and exceed effective context capacity.GLM-4.7 is cited as an example of degradation at overly large K.
6.5. Data Flywheel Simulation and Analysis
The simulated data flywheel iteratively updates the retriever from agent trajectories, producing steady gains in both agent success and retriever evidence recall.
- Imperfect agent trajectories can still provide useful supervision for iterative retriever improvement in open-ended search settings.
- Each iteration collects agent-interaction trajectories, updates the retriever, and evaluates the next retriever–agent interaction cycle.The simulation uses Tongyi-DeepResearch with 10K InfoSeekQA queries per step and the same agent for evaluation.
- Steady gains in success rate and evidence recall emerge across iterative updates.
7. Conclusion
The paper frames agent trajectories as supervision for retrieval in agentic search and introduces LRAT to convert trajectory behavior into training signals. Across diverse benchmarks and agent and retriever backbones, LRAT improves retrieval and end-to-end performance while supporting iterative retriever improvement.
- The paper identifies a fundamental misalignment between human-centric retrieval training and agentic search needs.
- Trajectory analysis finds browsing, unbrowsed results, and post-browse reasoning useful for estimating document utility and relevance intensity.
- LRAT converts agent trajectories into supervision signals for training agent-aligned retrievers.
- Experiments show LRAT is effective across in-domain and out-of-domain benchmarks, diverse agents, and retriever backbones.
- Agent trajectories can support iterative retriever improvement, indicating a potential sustainable data flywheel driven by agent interactions.