Source-linked AI summary
DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, Pengfei Liu
TL;DR
DeepResearcher addresses the gap between brittle prompt workflows or controlled RAG training and real-world web research. It trains LLM agents end-to-end with RL and live search interaction, reporting gains over both baseline types and emergent research behaviors. The paper also identifies contamination-related limits on generalization to scenarios whose answers are absent from training data.
Problem
Existing approaches use manually engineered prompts or controlled RAG environments, while open-source training datasets and disclosed real-web RL methods remain limited.
Method
DeepResearcher trains LLM agents end-to-end with reinforcement learning in genuine web environments and uses browsing agents to extract information from webpages.
Results
28.9 points improvement over prompt-engineered agents and up to 7.2 points over RAG-based RL agents are reported for research task completion.
Takeaways & Limitations
The results support direct interaction with real search environments as a crucial component for robust research capabilities in autonomous systems.
Takeaways & Limitations
Contamination can make models appear to benefit from web search while relying on memorized knowledge, limiting generalization when answers are absent from training data.
Abstract
from arXiv · showhide
Large Language Models (LLMs) equipped with web search capabilities have demonstrated impressive potential for deep research tasks. However, current approaches predominantly rely on either manually engineered prompts (prompt engineering-based) with brittle performance or reinforcement learning within controlled Retrieval-Augmented Generation (RAG) environments (RAG-based) that fail to capture the complexities of real-world interaction. In this paper, we introduce DeepResearcher, the first comprehensive framework for end-to-end training of LLM-based deep research agents through scaling reinforcement learning (RL) in real-world environments with authentic web search interactions. Unlike RAG-based approaches that assume all necessary information exists within a fixed corpus, our method trains agents to navigate the noisy, unstructured, and dynamic nature of the open web. We implement a specialized multi-agent architecture where browsing agents extract relevant information from various webpage structures and overcoming significant technical challenges. Extensive experiments on open-domain research tasks demonstrate that DeepResearcher achieves substantial improvements of up to 28.9 points over prompt engineering-based baselines and up to 7.2 points over RAG-based RL agents. Our qualitative analysis reveals emergent cognitive behaviors from end-to-end RL training, including the ability to formulate plans, cross-validate information from multiple sources, engage in self-reflection to redirect research, and maintain honesty when unable to find definitive answers. Our results highlight that end-to-end training in real-world web environments is not merely an implementation detail but a fundamental requirement for developing robust research capabilities aligned with real-world applications. We release DeepResearcher at https://github.com/GAIR-NLP/DeepResearcher.
Emergent Planning and Cross Validation Behaviors
DeepResearcher exhibits planning and cross-validation behaviors during multi-hop research. It can adjust plans and verify an initially identified answer before finalizing.
- The planning question targets the Rialto Bridge as the famous bridge in the composer’s birthplace.
- The cross-validation question identifies Herschell Gordon Lewis as the pioneer rather than Leah Meyerhoff.
- DeepResearcher can make plans and dynamically adjust or merge steps during reasoning.
- DeepResearcher identifies the correct answer during its first tool call but verifies it through subsequent steps before committing.
1 Introduction
The introduction positions DeepResearcher as an end-to-end RL framework for training LLM agents in genuine web environments. It reports improvements over prompt-engineered and RAG-based RL agents alongside emergent research behaviors.
- The introduction frames existing open research as lacking disclosed comprehensive methods for RL scaling with real-world web search.
- DeepResearcher trains LLM agents with reinforcement learning to interact directly with live search engines in real-world web environments.
- The framework combines real-world web search, end-to-end training, outcome rewards, and a specialized multi-agent architecture for webpage information extraction.
- 28.9 points improvement in research task completion is reported over prompt-engineered agents, while improvement over RAG-based RL agents reaches 7.2 points.
- Qualitative analysis reports planning, cross-validation, reflection, and honesty when exact answers cannot be found.
2 Related Work
Related work distinguishes prompt-based and training-based search agents and contrasts local RAG environments with real-world web search. DeepResearcher combines RL with genuine web interaction to address limitations of controlled retrieval settings.
- Search-agent research is categorized by prompting or training and by local RAG or real-world web-search environments.
- Manual workflows improve search capabilities but rely on carefully designed procedures for interacting with external knowledge sources.
- SFT and RAG methods can incur computational overhead or depend on supervised signals that weaken generalization to unknown scenarios.
- Local RAG systems use fixed repositories, creating concerns about information timeliness, domain adaptability, storage efficiency, and restricted search spaces.
- Real-world web search provides unstructured and heterogeneous information but introduces API-rate, latency, and webpage-structure challenges.
- DeepResearcher uniquely combines reinforcement learning with direct interaction with live search engines to handle open-web variability and complexity.
3 Methodology
DeepResearcher trains an agent to iteratively reason, search, browse, extract information, and answer questions in dynamic web environments. Its methodology combines GRPO, observation masking, outcome rewards, and infrastructure for reliable large-scale web interaction.
- 3 Methodology: The agent iteratively performs reasoning and tool selection based on the user question and observations.
- 3 Methodology: The web-search workflow returns structured titles, URLs, and snippets, while the browsing agent processes webpage segments sequentially using query-specific memory.
- 3.2 Addressing Challenges in Dynamic Real-World Web Environments: A 50-node distributed CPU cluster manages the large volume of search and webpage-crawling requests generated during RL rollouts.
- 3.2 Addressing Challenges in Dynamic Real-World Web Environments: Retry mechanisms address anti-crawling responses, failed requests, and provider rate limits encountered during web crawling and API use.
- 3.2 Addressing Challenges in Dynamic Real-World Web Environments: A dedicated reading agent partitions lengthy webpages, processes segments sequentially, and skips likely unproductive pages to improve extraction efficiency and accuracy.
- 3.3 RL Training Framework: GRPO optimizes the current policy using a reference policy and rollouts from an existing policy, estimating baselines without a separate critic.
- 3.3 RL Training Framework: Observation masking excludes tool outputs from training so that only the model’s responses contribute to optimization.
- 3.4 Reward: F1 is the primary reward metric for short-answer open-domain QA; incorrect formatting receives -1, while correctly formatted answers receive word-level F1 rewards.
4 Beyond Memorization: Curating Search-Dependent Training Data
DeepResearcher curates search-dependent training data to prevent memorization from masquerading as web-search ability. It filters unreliable questions and contamination, producing a dataset weighted toward multi-hop research.
- No open-source training dataset specifically targets deep research, so the framework repurposes open-domain QA datasets requiring online search.
- Data contamination matters because pretrained models may already contain benchmark answers, making apparent search success misleading.
- Contamination can falsely suggest search benefits, weaken genuine strategy learning, and impair generalization to novel questions.
- The pipeline removes time-sensitive, subjective, and harmful questions using DeepSeek-R1-based quality filtering.
- Contamination detection samples 10 base-model responses per question and excludes questions answered correctly without search.
- 80,000 final examples follow a 1:1:3:3 NQ:TQ:HotpotQA:2Wiki ratio, emphasizing multi-hop cases at 75%.
5 Experiments
Training uses Qwen2.5-7B-Instruct with large-scale rollout sampling, allowing each trajectory to interact with tools repeatedly before answering.
- DeepResearcher trains Qwen2.5-7B-Instruct with 256 prompts per step and 16 rollouts per prompt.
- Each rollout permits up to 10 tool calls followed by a final answer, with a mini-batch size of 4,096.
5.2 Evaluation and Results
The evaluation covers in-domain and out-of-domain open-domain QA, using multiple datasets and sampled development examples to test robustness and generalization against varied baselines.
- Evaluation datasets: In-domain evaluation uses NQ, TriviaQA, HotpotQA, and 2Wiki development sets.
- Evaluation datasets: Out-of-domain evaluation adds MuSiQue, Bamboogle, and PopQA to test generalization beyond the training domain.
- Sampling: The benchmark samples 512 development examples from six datasets and includes all 125 Bamboogle examples.
- Baselines: Baselines span unaided Chain-of-Thought, RAG, snippet-based Search-o1, web-enabled Search-o1, and Wikipedia-retrieval RL agents.
- Baselines: R1-Searcher uses Bing but restricts queries to Wikipedia and summarizes the first three results, unlike DeepResearcher’s unrestricted URL selection.
5.2 Evaluation and Results
DeepResearcher is evaluated with F1 and model-based judgment across in-domain and out-of-domain tasks. It leads broadly, especially when research requires navigating information beyond fixed corpora.
- In-domain results: DeepResearcher leads in-domain MBE across four datasets and remains competitive in F1, especially on TriviaQA and 2Wiki.
- Search behavior: The framework allows autonomous URL selection rather than requiring summaries of the top three search results.
- Metrics: Rule-based evaluation uses normalized-answer F1, while long-form responses additionally receive GPT-4o-mini correctness judgments.
- Real-world environment: Bamboogle requires knowledge beyond Wikipedia, where DeepResearcher substantially outperforms local RAG methods.
- Out-of-domain results: DeepResearcher leads both F1 and MBE on all three out-of-domain datasets, demonstrating strong generalization.
6 Analysis
DeepResearcher’s reinforcement-learning training progressively increases performance, tool use, and response length, while qualitative cases show planning, cross-validation, reflection, and honesty behaviors.
- Training Dynamics: F1 rises from 0.375 to around 0.55 as reinforcement-learning training progresses.
- Training Dynamics: Tool calls increase across reasoning difficulties during training, with the 4-hop setting continuing to rise after 34 steps.The trend suggests continued information retrieval for harder questions.
- Training Dynamics: Response lengths increase with reasoning complexity and continue rising across all four settings without saturation.The outputs include more double-checking, refinement, and planning.
- Planning: DeepResearcher forms and dynamically adjusts plans, including merging steps when appropriate, without explicit planning-data SFT.
- Cross-validation: DeepResearcher cross-validates an answer found in its first tool call instead of immediately committing to it.The paper describes this cautious verification as improving response reliability and robustness.
- Reflection and Honesty: When retrieved information conflicts with a question, DeepResearcher refines its search query; when it lacks the answer, it declines to respond.These cases illustrate reflection and honesty, although current question-answering metrics do not account for honesty.
7 Conclusion
DeepResearcher scales reinforcement learning for LLM agents in live web-search environments rather than static or controlled retrieval settings. It reports improved task completion and research capabilities, alongside planning, cross-validation, reflection, and honesty behaviors.
- DeepResearcher trains LLM agents to operate directly in real-world web-search environments and navigate the open web’s complexity and variability.
- DeepResearcher reports substantial improvements in task completion and research capabilities over prompt-engineered and RAG-based RL agents.The conclusion characterizes direct engagement with dynamic search environments as central to these improvements.
- The end-to-end framework addresses web-search challenges and enables agents to develop problem-solving strategies autonomously through a multi-agent architecture.The system handles network latency and anti-crawling mechanisms while collecting diverse web information.
A Prompts
The prompts specify quality checks, answer evaluation, and a structured web-research workflow with explicit formats for reasoning, tool calls, and final answers.
- A.1 Prompt for Question Quality Level Evaluation: Question-quality evaluation classifies questions as time-sensitive, harmful, subjective, or good.The prompt inserts each question into a placeholder and requires one corresponding answer tag.
- A.1 Prompt for Question Quality Level Evaluation: Contamination detection asks the model for a short English answer to test whether responses may reflect training-data contamination.The appendix labels this as a contamination-detection prompt and specifies short English responses.
- A.2 Prompt for Model’s Answer Quality Level Evaluation: Answer-quality evaluation compares a predicted answer with a ground-truth answer list and accepts semantic equivalence to at least one listed answer.The evaluator first provides a rationale and then returns a correct-or-incorrect judgment in JSON format.
- A.2 Prompt for Model’s Answer Quality Level Evaluation: The model-based evaluation prompt supplies the question, ground-truth answers, and predicted answer as inputs to the judgment procedure.Its required output contains a rationale and a judgment restricted to correct or incorrect.
- A.3 Prompt for Research Plan on Question Answering: The research-plan prompt frames each task as complex deep research requiring web search and webpage browsing.The prompt identifies the assistant and provides the current date as background information.
- A.3 Prompt for Research Plan on Question Answering: Before answering, the model is instructed to determine a research plan or what to search next.The output format separates thinking, tool calls, and the final answer, and requires strict adherence to those formats.
- A.3 Prompt for Research Plan on Question Answering: Final responses must appear inside answer tags without explanations or extra information, while intermediate reasoning and tool calls use their designated formats.Yes-or-no questions receive only yes or no in the final answer.
- A.3 Prompt for Research Plan on Question Answering: Tool calls must use structured XML-wrapped JSON containing the function name and arguments.The available functions are web search and webpage browsing, with specified query and URL-list parameters.
B Training Scaling Result
Training-scale monitoring reports F1 scores across seven benchmarks, with DeepResearcher scaling across all of them and especially strongly on out-of-domain benchmarks.
- B Training Scaling Result: F1 scores were tracked on seven benchmarks using 125 sampled development-set cases per benchmark.The reported training curves assess scaling across all seven benchmarks.
- B Training Scaling Result: DeepResearcher can scale in all seven benchmarks, especially in out-of-domain benchmarks.Figure 7 presents the corresponding F1 score during training.