DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments

Emergent Planning and Cross Validation Behaviors

• DeepResearcher forms plans and dynamically adjusts them throughout the reasoning process.It can also merge steps when appropriate.Source: Figure 2 · p. 2
• After identifying the correct answer in its first tool call, DeepResearcher verifies its accuracy through subsequent steps instead of immediately committing.The figure describes this verification as enhancing the reliability and robustness of final answers.Source: Figure 2 · p. 2

• Unlike static-corpus RAG, DeepResearcher trains in noisy, dynamic web environments where information may be absent or outdated and search quality and webpage formats vary.Source: §1 · p. 3
• The framework scales outcome-reward RL end-to-end without human priors, training agents to search live engines and synthesize diverse web information for open-domain questions.Source: §1 · pp. 3–4
• The system addresses real-web training challenges including API rate limits, network latency, anti-crawling mechanisms, and webpage variability, using browsing agents to extract relevant information from full webpages.Source: §1 · pp. 3–4
• DeepResearcher improves research task completion by up to 28.9 points over prompt-engineered agents and up to 7.2 points over RAG-based RL agents.Source: §1 · p. 3
• Qualitative analysis finds that end-to-end RL training yields planning, cross-validation across sources, reflection to redirect research, and honesty when exact answers are unavailable.Source: §1 · pp. 3–4
• The authors release the complete DeepResearcher training framework as open source.Source: §1 · p. 4

2.1 Prompt-Based Search Agents

• Prompt-based agents use human-engineered workflows that can constrain adaptability, despite progress in systems such as OpenResearcher (Zheng et al., 2024) and Search-o1 (Li et al., 2025a).The cited work also includes AirRAG, IterDRAG, Plan*RAG, and Open Deep Search.Source: §2.1 · p. 4

2.2 Training-Based Search Agents

• SFT offers an alternative to manual RAG workflow optimization, but CoRAG (Wang et al., 2024b) incurs MCTS overhead and may generalize poorly beyond supervised scenarios.CoRAG uses Monte Carlo Tree Search to select document blocks under budget constraints.Source: §2.2 · p. 4
• RL search agents such as Search-R1 (Jin et al., 2025), ReSearch (Chen et al., 2025), and R1-Searcher (Song et al., 2025) develop retrieval reasoning without manually specified cues.OpenAI has acknowledged using RL to enhance its research agent, but its methods remain proprietary and undisclosed.Source: §2.2 · p. 4

2.3 Training Environments

• Local RAG relies on fixed repositories, which can suffer from stale information, poor domain adaptability, and storage bottlenecks; its RL evaluation remains tied to predefined knowledge bases and similarity search.The passage names Search-R1 (Jin et al., 2025), ReSearch (Chen et al., 2025), and R1-Searcher (Song et al., 2025) as RAG-based RL approaches.Source: §2.3 · pp. 4–5
• Real-world web search provides real-time, heterogeneous information but brings API-rate-limit, network-latency, and webpage-structure challenges, and is seldom trained end-to-end.These environments more closely reflect the complexity of actual research tasks than fixed corpora.Source: §2.3 · p. 5
• DeepResearcher trains agents to interact directly with live search engines, using RL to learn adaptive strategies for the open web’s variability and complexity.This contrasts with prior RL methods focused primarily on static local corpora and aims to address limitations of both prompt-based and RAG-confined approaches.Source: §2.3 · p. 5

3.1 Deep Research Trajectory

• DeepResearcher iteratively reasons about the user question and observations, then selects tools and issues JSON-formatted web searches returning structured results with a fixed top-k.Reasoning precedes action, and final responses are enclosed in <answer> tags.Source: §3.1 · p. 5, §3.2 · p. 6

3.2 Addressing Challenges in Dynamic Real-World Web Environments

• A browsing agent maintains query-specific short-term memory, reads webpage segments sequentially, decides whether to continue, and returns relevant information to the researcher.The multi-agent approach can skip unproductive pages when their initial segments are predominantly irrelevant.Source: §3.2 · p. 6
• A 50-node CPU cluster handles high-volume search and crawling requests during RL rollouts, while retries and cached search results mitigate crawling failures and API limits.Identical queries can use cached results for a predetermined period, such as 7 days.Source: §3.2 · p. 6

3.3 RL Training Framework

• GRPO optimizes the current policy using a reference policy and rollouts from an existing policy, estimating its baseline from group trajectories instead of a separate critic.The method adopts Group Relative Policy Optimization (GRPO).Source: §3.3 · p. 6

3.4 Reward

• Tool observations are masked from training; on short-answer QA, correctly formatted answers receive a word-level F1 reward, while incorrect formatting incurs -1.F1 is the primary reward metric for the open-domain QA datasets.Source: §3.4 · p. 7

4.1 Leveraging Open Domain QA Data

• The corpus draws on NQ and TriviaQA for single-hop questions, and HotpotQA and 2WikiMultiHopQA for multi-hop questions requiring information from multiple sources.The multi-hop datasets were designed to evaluate multi-step reasoning.Source: §4.1 · p. 7

4.2 The Issue of Data Contamination

• Pretraining may expose models to benchmark answers, letting them appear to benefit from search while relying on memorized knowledge and weakening performance on novel research questions.The paper argues contamination can impede genuine search-strategy development and generalization when answers are absent from training data.Source: §4.2 · p. 7

4.3 Data Cleaning and Contamination Detection

• Quality filtering removes time-sensitive, highly subjective, and potentially harmful or policy-violating questions, using DeepSeek-R1 (Guo et al., 2025) to identify them.The filtering is intended to avoid questions that could produce unreliable or problematic search results.Source: §4.3 · p. 7
• To detect contamination, the base model generates 10 responses per question; questions with a ground-truth answer in any response are excluded.This pass@10 screening aims to prevent apparent search success from relying on memorized parametric knowledge.Source: §4.3 · p. 8
• 80,000 examples form the final dataset, with a 1:1:3:3 NQ:TQ:HotpotQA:2Wiki ratio that weights multi-hop questions at 75%.The authors describe this distribution as reflecting the complex information-seeking behaviors required for deep research.Source: §4.3 · p. 8

5.1.1 Model and Hyperparameters

• Training uses Qwen2.5-7B-Instruct with 256 prompts per step, 16 rollouts per prompt, up to 10 tool calls per rollout, and a mini-batch size of 4,096.The pipeline uses the verl framework; one rollout stage backpropagates once.Source: §5.1.1 · p. 8

5.2.1 Benchmarks

• In-domain evaluation uses the dev sets of NQ, TQ, HotpotQA, and 2Wiki.The passages cite Kwiatkowski et al. (2019), Joshi et al. (2017), Yang et al. (2018), and Ho et al. (2020), respectively.Source: §5.2.1 · p. 8
• Out-of-domain evaluation adds MuSiQue, Bamboogle, and PopQA to test generalization beyond the training domain.The passages cite Trivedi et al. (2022), Press et al. (2022), and Mallen et al. (2022), respectively.Source: §5.2.1 · p. 8
• Evaluation samples 512 development examples from NQ, TQ, HotpotQA, 2Wiki, MuSiQue, and PopQA, plus all 125 Bamboogle examples.The stated purpose is to assess robustness across topics and reasoning requirements.Source: §5.2.1 · p. 8

5.2.2 Baselines

• Baselines include CoT Only without external context, RAG with retrieved context, and Search-o1 with retrieved snippets rather than full documents.Search-o1 was reimplemented with the authors’ own prompt for consistency with other results.Source: §5.2.2 · p. 8
• Search-o1 + Web Search can issue real-time API queries and browse URLs, enabling richer, more dynamic information gathering than snippet-limited Search-o1.The passage presents this setting as the basis of deep research.Source: §5.2.2 · p. 8
• Search-r1 trains and infers using retrieved Wikipedia information, while R1-Searcher sends Bing queries restricted to en.wikipedia.org and summarizes the first three result pages.Search-r1 has base and instruct variants based on the initial actor model; the supplied R1-Searcher comparison also says DeepResearcher trains in a real-world environment and does not restrict its search domain to Wikipedia.Source: §5.2.2 · p. 8

5.2.4 Main Results

• DeepResearcher has the highest in-domain MBE performance across four datasets, with competitive F1 and a more challenging web-search setting than local-RAG Search-r1-base.It particularly excels on TQ and 2Wiki; Search-r1-base has comparable MBE results on NQ and HotpotQA but directly accesses the relevant Wikipedia corpus.Source: Table 1 · p. 9, §5.2.4 · p. 9

5.2 Evaluation and Results

• The method lets the model select URLs autonomously instead of requiring it to summarize the top three search results.Source: §5.2 · p. 9

5.2.3 Evaluation Metrics

• F1 is computed after lowercasing answers and removing punctuation, while MBE uses GPT-4o-mini judgments of correctness against each question and ground truth.MBE is the accuracy of the model-based correct-or-incorrect judgments.Source: §5.2.3 · p. 9

5.2.4 Main Results

• On Bamboogle, which requires knowledge beyond Wikipedia’s coverage, DeepResearcher significantly outperforms local-RAG methods and R1-Searcher using real-web search.The results support the advantage of real-world environments during RL scaling for developing robust information retrieval and synthesis capabilities.Source: §5.2.4 · p. 9
• DeepResearcher leads all baselines in both F1 and MBE across the three out-of-domain datasets, demonstrating generalization beyond its training domains.The datasets are Musique, Bamboogle, and PopQA.Source: §5.2.4 · p. 9, Table 2 · p. 10

6.1 Training Dynamics

• F1 rose from 0.375 to around 0.55 as reinforcement learning training progressed.The reported evaluation shows a consistent upward trend.Source: Figure 4 · p. 10, §6.1 · p. 10
• As training advances, tool-call turns rise across difficulty levels—with 4-hop still increasing after 34 steps—and response lengths grow across all four hop settings.Longer responses reflect continued expansion of reasoning, including double-checking, refinement, and planning.Source: Figure 4 · p. 10, §6.1 · p. 10

6.2 Case Study

• DeepResearcher dynamically adjusts multi-hop plans, merges steps when appropriate, and exhibits planning without SFT on explicit planning data (Yue et al., 2024a).Source: Figure 5 · p. 11, §6.2 · p. 13
• Even after finding the correct answer on its first tool call, DeepResearcher verifies it in subsequent steps before finalizing its response.Source: Figure 5 · p. 11, §6.2 · p. 13
• When retrieved information conflicts with the question, DeepResearcher uses environmental feedback to refine its search query, helping prevent reasoning dead ends (Fu et al., 2025).Source: Figure 6 · p. 12, §6.2 · p. 13
• When it has not found the correct answer, DeepResearcher declines to respond, although current question-answering metrics do not account for this reliability behavior.Source: Figure 6 · p. 12, §6.2 · p. 13

7 Conclusion

• DeepResearcher’s direct interaction with live search engines improves task completion and research capabilities over prompt-engineered and RAG-based RL agents.Unlike static knowledge bases or controlled retrieval settings, it trains agents to navigate the complexity and variability of the open web.Source: §7 · p. 13
• End-to-end training lets agents develop problem-solving strategies autonomously, while a robust multi-agent architecture helps collect diverse web information despite latency and anti-crawling challenges.The resulting system demonstrates planning, cross-validation, reflection, and honesty.Source: §7 · p. 13
• Scaling reinforcement learning in real-world environments can improve research performance and offers a path toward adaptive systems for complex, open-domain problems.The paper presents this as relevant to real-world applications.Source: §7 · p. 13

A.1 Prompt for Question Quality Level Evaluation

• Question screening distinguishes time-sensitive questions, whose answers may change, harmful or offensive questions, and subjective questions not based on facts.Questions outside those categories are labeled good.Source: §A.1 · p. 16
• The screening prompt inserts the question into a placeholder and requires one of four labels inside answer tags: time sensitive, harmful, subjective, or good.The good label applies when none of the listed categories applies.Source: §A.1 · p. 16
• The contamination-detection prompt asks for a short English answer to a question to test whether AI responses are influenced by training-data contamination.It includes a question placeholder followed by a response field.Source: §A.1 · p. 16

A.2 Prompt for Model’s Answer Quality Level Evaluation

• Answer evaluation requires a rationale and a correct-or-incorrect judgment based on whether the prediction is semantically equivalent to at least one ground-truth answer.The predicted answer need not match a ground-truth answer verbatim.Source: §A.2 · p. 17

A.3 Prompt for Research Plan on Question Answering

• For complex questions, the research prompt instructs the assistant to plan what to research next before answering, using web search and webpage browsing tools.It frames the task as deep research and says the assistant need not answer immediately.Source: §A.3 · p. 18
• The research prompt specifies separate formats for thinking followed by an answer and for issuing a tool call, with the required XML-style tags.The tool-call format includes a function name and arguments.Source: §A.3 · p. 18

• DeepResearcher scales on all seven benchmarks, especially OOD benchmarks, with F1 tracked during training.The evaluation sampled 125 cases from each benchmark’s development set.Source: Figure 7 · p. 19, B Training Scaling Result · p. 19