BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent

Introduction

• Current evaluations conflate agent performance with retrieval effectiveness, while dynamic, black-box web APIs hinder reproducibility and obscure retrieval-model contributions.Because those APIs search across a changing internet corpus, retrieval cannot be controlled consistently.Source: §1 · pp. 1–2
• Commercial web-search APIs add substantial costs and retrieval-quality variability, limiting accessibility and introducing uncertainty into benchmarking.Source: §1 · p. 2
• BrowseComp-Plus extends BrowseComp [1] with a fixed, human-verified corpus of supporting and hard-negative documents, enabling independent evaluation of retrieval and agent components.The benchmark is designed to reduce costs and improve reproducibility and robustness by eliminating reliance on dynamic web APIs.Source: §1 · pp. 2–3
• Systematic evaluations of broad retriever and LLM combinations find that even state-of-the-art retrievers struggle to surface all evidence needed for reasoning-intensive queries.The controlled setup supports analysis of retrieval–agent interactions and helps identify performance bottlenecks.Source: §1 · pp. 2–3
• On original BrowseComp queries, state-of-the-art retrieval models also struggle to retrieve relevant documents for complex, reasoning-intensive queries.This analysis was enabled by a fixed corpus and grounded relevant-document judgments.Source: §1 · pp. 2–3
• The authors release benchmark data, evaluation scripts, and baselines to support reproducible research on deep-research systems.Source: §1 · p. 3

2.1 Deep-Research Agent

• Deep-research agents use iterative query reasoning, search planning, and reflection [3], outperforming single-round RAG [2]; BrowseComp-Plus enables fair comparison using a fixed retriever.Search R1 [12] and WebSailor [13], built on Qwen [14], use reinforcement learning to enhance search-tool capabilities.Source: §2.1 · p. 3

2.2 Neural Retrieval

• Neural retrieval encodes queries and documents as dense vectors for nearest-neighbor search [15] [16], improving over BM25, while retriever contributions to agent performance remain underexplored.Advances include continuous pretraining [19], data augmentation [20] [21] [22], LLM backbones [23] [24], and distillation [25] [26]; BrowseComp-Plus enables systematic retriever evaluation.Source: §2.2 · p. 3

2.3 Deep Retrieval Benchmarks

• Traditional retrieval benchmarks are mostly single-hop or Wikipedia-limited, whereas BrowseComp [1] targets complex search and ZH-BrowseComp [4] and MedBrowseComp [5] extend its scope.Existing benchmarks generally lack standardized corpora; BrowseComp-Plus supports fair evaluation with a human-verified corpus.Source: §2.3 · pp. 3–4

3 BrowseComp-Plus

• BrowseComp-Plus builds on BrowseComp [1] to support independent evaluation of retrieval and LLM components within deep-research systems.BrowseComp provides challenging fact-seeking questions with concise, verifiable answers, but its question-answer-only format complicates isolated measurement of retrieval effectiveness.Source: §3 · p. 4, §3.1 · p. 4

3.1 Preliminary: BrowseComp

• BrowseComp contains 1,266 difficult fact-seeking questions with concise, verifiable answers, but its end-to-end format complicates isolated measurement of retrieval effectiveness [1].Source: §3.1 · p. 4

3.2 Building the Document Corpus

• Corpus construction targeted complete evidence coverage, retrieval difficulty from distracting negatives, and practical size, using automated evidence mining followed by human verification.The resulting corpus supports study of the deep-research framework.Source: §3.2 · p. 4

3.2.1 Evidence Document Gathering

• OpenAI o3 failed to provide evidence for 124 of 1,266 pairs, and scraping exclusions removed 137 of the remaining 1,142, leaving 1,005 for verification.The model searched for clue-level supporting pages, which were scraped and parsed for human review.Source: §3.2.1 · pp. 4–5

3.2.2 Evidence Document Verification

• Annotators marked document spans supporting each clue, checked that the evidence covered the full answer, and labeled documents that contained the answer, including implicitly.If initial evidence failed either verification criterion, annotators revised clues and searched for additional documents for at least 20 minutes.Source: §3.2.2 · p. 5
• 830 of the 1,005 candidate pairs passed human verification, with over 80% average annotator agreement.Fourteen university student annotators spent over 400 hours labeling; gold documents were identified semantically, not just by exact answer-string matches.Source: §3.2.2 · pp. 5–6

3.3 Hard Negative Mining

• GPT-4o split each question into about seven sub-queries, each sent to SerpAPI for up to 100 results that were scraped as hard negatives.The negatives were mined to challenge retrieval while keeping the corpus a practical size.Source: §3.3 · p. 6

3.4 Final Corpus Statistics

• 100,195 documents and 830 queries form the corpus, averaging 6.1 evidence documents, 76.28 negatives, and 2.9 gold documents per query.Documents average 5,179.2 words and 32,296.2 characters.Source: §3.4 · p. 6

4.1 Baselines: LLM Search Agents; 4.2 Baselines: Retriever; 4.3 Experiment Setup

• The study evaluates commercial and open-source language models for agentic search, providing a retriever tool and using the BrowseComp-derived prompt across models except Search-R1, which uses its fine-tuning-aligned prompt.The retriever returns the top k = 5 results, each truncated to 512 tokens; 86.5% of queries still contain the ground-truth answer in at least one gold document.Source: §4.1 · p. 6, §4.3 · p. 7

4.4 Evaluation Metrics

• End-to-end evaluation reports accuracy, evidence-document recall, search calls, and calibration error; standalone retrieval uses Recall@k and nDCG@k against human-verified evidence and gold documents.Calibration error is not reported for Search-R1 because its fixed input/output format provides no confidence estimate.Source: §4.4 · pp. 7–8

4.6 End-to-End Deep-Research Performance

• 70.12% accuracy: gpt-5 with Qwen3-Embedding-8B leads, while gpt-oss-120B in high reasoning mode reaches 42.89% and outperforms Opus 4 with the same retriever.With Qwen3-Embedding-8B, Qwen3-32B scores 10.36%, gpt-4.1 35.42%, and o3 63.49%; open-source agents also make fewer than 2 search calls per query, versus more than 20 for gpt-5 and o3.Source: Table 1 · p. 8, §4.6 · p. 9

4.7 Effect of Retrieval Quality

• Stronger retrievers improve agent accuracy and reduce search overhead, while Qwen3-Embedding-8B and ReasonIR-8B outperform BM25 on retrieval metrics.Qwen3-Embedding scales with model size, with Qwen3-8B surpassing ReasonIR-8B at 8B; Qwen3-Embedding-8B improves gpt-5 accuracy from 55.9% to 70.12% and reduces calls by about 1–3 for most proprietary models. BM25 agents also achieve lower citation metrics despite more calls.Source: Table 2 · p. 9, §4.7 · pp. 9–10, Table 3 · p. 10

4.8.1 Oracle Retrieval

• 93.49% accuracy: gpt-4.1 answers with oracle positive documents, versus 14.58% with BM25; Qwen3-32B reaches 83.25% in the oracle setting.The 6.51% of gpt-4.1 oracle errors were confirmed answerable from the positive documents and attributed to reasoning failures; 50 Qwen3-32B errors (6%) arose because positive documents exceeded its context window.Source: §4.8.1 · pp. 10–11

4.8.2 Impact of Reasoning Effort

• Higher reasoning effort improves accuracy and recall but increases search calls, while calibration error tends to decrease.For oss-20b with Qwen3-Embed-8B, accuracy rises from 13.37% to 34.58% and recall from 17.37% to 49.29%; calls increase from about 2 to 24. Qwen3-Embed-8B beats BM25 across reasoning settings.Source: §4.8.2 · p. 11

4.8.3 Effect of Document Reading Strategy

• A whole-document reader raises gpt-4.1 accuracy from 35.42% to 43.61%, but Qwen3-32B improves only from 10.36% to 11.69%.The tool adds 1.85 full-document reads per gpt-4.1 query and 0.27 get-document calls per Qwen3-32B query; gains depend on agents recognizing when and how to use it.Source: §4.8.3 · p. 12

4.8.4 Effect of Corpus Size

• Expanding the corpus about tenfold to 9,771,311 documents does not change the overall conclusions about retriever or agent rankings.BM25 retrieval improves across metrics, whereas Qwen3-Embedding-8B and ReasonIR-8B retrieval scores fall as unjudged Fineweb documents enter top ranks; Qwen3-32B with Qwen3-Embedding-8B drops from 10.36% to 7.11% accuracy.Source: §4.8.4 · pp. 12–13

5 Future Work and Discussion

• BrowseComp-Plus aims to play for Deep-Research studies a role like NQ and HotpotQA [30] played for modern neural QA, supporting design, comparison, and diagnosis.It retains BrowseComp’s challenge while providing a more controlled and transparent experimental setup.Source: §5 · p. 13
• How retriever quality affects agent learning remains open; optimization may be harder with BM25 because it surfaces fewer relevant documents than modern embedding-based retrievers.The current work studies how retrievers influence inference performance, rather than their role during agent optimization.Source: §5 · p. 13
• A future direction is testing whether an agent optimized with BM25 generalizes when switched to an embedding-based search tool.Source: §5 · p. 13
• Future work could test whether an LLM can orchestrate tiered search tools or sub-agents for federated search [42] and results comparable in quality to Google.The discussion frames this as a creative attempt to break down commercial search engines’ composed, multi-facet solutions.Source: §5 · p. 13
• Retrieval models could be made tolerant of or adaptive to a specific agent, and co-optimized with it for answer accuracy rather than developed in isolation.In Deep Research, the primary consumer of retrieved documents is a tool-augmented LLM agent, not a human.Source: §5 · p. 13
• An oracle retriever that surfaces gold or highly relevant documents can improve accuracy and may reduce search iterations, motivating high-precision retrieval for complex queries.The authors suggest this could benefit real-world applications.Source: §5 · p. 13
• BrowseComp-Plus is presented as a testbed for systematic, fine-grained analysis of agent–retriever interactions in Deep Research.Source: §5 · p. 13

6 Conclusion

• BrowseComp-Plus addresses reproducibility, fairness, and transparency with a fixed, human-verified corpus of positive and hard-negative documents for controlled component assessment.The framework enables independent assessment of retrieval and agent components.Source: §6 · p. 14
• The benchmark supports future work on co-optimizing retrievers and agents, improving out-of-distribution tool-use generalization, and advancing context engineering.The authors aim for BrowseComp-Plus to provide a robust platform for probing these dynamics.Source: §6 · p. 14
• The authors aim to catalyze the next generation of deep-research systems by making the benchmark and baselines publicly available.Source: §6 · p. 14

A OpenAI O3 Evidence Document Gathering Prompt

• The O3 evidence-gathering prompt asks for online evidence supporting a given answer, listed step by step with URLs ordered for human verification.Source: A OpenAI O3 Evidence Document… · p. 22
• The prompt requires a three-column table listing each question clue, its evidence URL, and the page content supporting that clue.Source: A OpenAI O3 Evidence Document… · p. 22

• Figure 5 shows a screenshot of the annotation interface.Source: Figure 5 · p. 22

C Problematic Cases

• Some BrowseComp question-answer pairs were flawed because a book’s writer and the person who later returned to acting shared a name but were different people.Source: C Problematic Cases · p. 23
• 42 distance-related queries requiring multiple Google Maps calls were removed because static corpus documents cannot provide the intended API-based capability.Static snapshots of Maps pages were not considered a valid substitute for access to the Google Maps API.Source: C Problematic Cases · p. 23
• 13 queries with ambiguous answer formats or multiple valid answers were removed, including a case where three usernames were credited as story authors.Source: C Problematic Cases · p. 23

D Negative Mining Query Decomposition Prompt

• The negative-mining prompt asks an expert to decompose each complex, multi-part question into smaller subqueries that collectively provide enough information to answer it.Source: D Negative Mining Query Decomposition… · p. 23
• Each subquery must address one piece of information, be independently answerable without pronouns or references, and retain necessary context and constraints from the original query.Source: D Negative Mining Query Decomposition… · p. 23
• The prompt requires preserving all important details and returning only the subqueries as a JSON array of strings.Source: D Negative Mining Query Decomposition… · p. 23

• The agent should reason step by step while interacting with the search engine and may use the search tool multiple times.Source: E Main Search Prompt · p. 24
• The required response includes an explanation whose evidence documents are cited inline using their docids in square brackets.The prompt gives [20] as an example citation format.Source: E Main Search Prompt · p. 24
• The response must also provide a succinct exact answer and a confidence score between 0% and 100%.Source: E Main Search Prompt · p. 24

F Evaluation Prompt

• The evaluator extracts the response’s final exact answer, using “None” when no exact final answer is present.It also records the response’s confidence score, defaulting to 100 when none is available.Source: F Evaluation Prompt · p. 24
• The evaluation reasoning should explain only meaningful differences between the extracted answer and correct answer, without solving the problem or arguing for another answer.The prompt frames evaluation as a judgment of the response against the supplied precise and unambiguous correct answer.Source: F Evaluation Prompt · p. 24
• Correctness is “yes” only when the extracted answer matches the correct answer or falls within a small numerical margin; otherwise it is “no.”Inconsistency, ambiguity, non-equivalence, or an incorrect answer warrants “no.”Source: F Evaluation Prompt · p. 24

G Search Prompt with Get-Doc

• The search agent must use the provided search and get_document tools iteratively, interleaving tool use with reasoning and repeating calls as needed.Source: G Search Prompt with Get-Doc · p. 25
• The required response has an Explanation with inline evidence-document IDs in square brackets, followed by an Exact Answer and a 0%–100% Confidence score.The example places document IDs at the ends of explanation sentences.Source: G Search Prompt with Get-Doc · p. 25

H API Cost

• Table 8 presents the API costs of the experiments.Source: H API Cost · p. 25