Source-linked AI summary
BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, Jimmy Lin
TL;DR
Existing deep-research benchmarks rely on dynamic black-box search, limiting fair, reproducible, and component-focused evaluation. BrowseComp-Plus introduces a fixed corpus with human-verified evidence and hard negatives, and its experiments show that retrieval quality affects both effectiveness and efficiency while substantial progress remains possible.
Problem
Black-box web search and uncontrolled corpora limit fair comparisons, reproducibility, and isolation of retriever contributions in deep-research evaluation.
Method
BrowseComp-Plus uses a fixed, human-verified corpus with supportive and hard-negative documents to evaluate retrieval and agent components independently.
Results
Retrieval quality substantially impacts deep-research effectiveness and efficiency, with stronger retrievers improving accuracy and reducing required search iterations.
Takeaways & Limitations
BrowseComp-Plus provides a controlled platform for diagnosing retrieval–agent dynamics and studying future co-optimization and context engineering.
Takeaways & Limitations
Retriever effects during agent optimization remain an open question, particularly because BM25 may surface fewer relevant documents than embedding-based retrievers.
Abstract
from arXiv · showhide
Deep-Research agents, which integrate large language models (LLMs) with search tools, have shown success in improving the effectiveness of handling complex queries that require iterative search planning and reasoning over search results. Evaluations on current benchmarks like BrowseComp relies on black-box live web search APIs, have notable limitations in (1) fairness: dynamic and opaque web APIs hinder fair comparisons and reproducibility of deep research methods; (2) transparency: lack of control over the document corpus makes it difficult to isolate retriever contributions. In other words, the current evaluations may compare a complete deep research system at a given time, but they do not foster well-controlled experiments to provide insights into the capability of underlying deep research LLMs. To address these challenges, we introduce BrowseComp-Plus, a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus. Each query in BrowseComp-Plus includes human-verified supporting documents and mined challenging negatives, enabling controlled experimentation. The benchmark is shown to be effective in distinguishing the performance of deep research systems. For instance, the open-source model Search-R1, when paired with the BM25 retriever, achieves 3.86% accuracy, whereas the GPT-5 achieves 55.9%. Integrating the GPT-5 with the Qwen3-Embedding-8B retriever further enhances its accuracy to 70.1% with fewer search calls. This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods, fostering insights into retrieval effectiveness, citation accuracy, and context engineering in Deep-Research system.
1 Introduction
BrowseComp-Plus addresses fairness, transparency, and accessibility problems in deep-research evaluation by fixing the document corpus and separating retrieval from agent performance. Its controlled setup supports systematic analysis of retrieval–agent interactions and reveals persistent retrieval challenges.
- Current benchmarks conflate agent performance with retrieval effectiveness, hindering fair comparisons and reproducibility.
- Black-box web search APIs are dynamic, opaque, costly, and variable, limiting transparency and accessibility.
- BrowseComp-Plus uses a fixed, human-verified corpus containing supportive and hard-negative documents for independent component evaluation.
- The benchmark enables systematic analysis of retrieval–agent combinations and identification of performance bottlenecks.
- State-of-the-art retrievers still struggle to surface all necessary evidence for reasoning-intensive queries.
- The released benchmark data, evaluation scripts, and baselines support reproducible research.
2 Related Works
Deep-research benchmarks target complex, iterative search and reasoning, but existing evaluations commonly lack standardized corpora and fixed retrievers. BrowseComp-Plus fills this gap while supporting controlled comparisons of retrieval methods and language-model agents.
- Deep-research agents use iterative query reasoning, search planning, and reflection, extending beyond single-round retrieval-augmented generation.
- Fair agent evaluation requires a fixed retriever, whereas existing studies mostly rely on black-box web search APIs.
- Neural retrievers use dense query and document vectors with nearest-neighbor search and have improved effectiveness over BM25.
- Retriever advances include continuous pretraining, data augmentation, LLM backbones, and distillation, but retriever contributions remain underexplored.
- Traditional benchmarks emphasize single-hop questions, while HotpotQA uses a limited Wikipedia corpus extensively covered during LLM training.
- BrowseComp addresses complex, hard-to-find information through questions with verifiable answers, but its integrated evaluations lack standardized corpora.
3 BrowseComp-Plus
BrowseComp-Plus constructs a controlled corpus for complex BrowseComp questions through automated evidence gathering, human verification, and hard-negative mining. The resulting dataset balances evidence coverage, retrieval difficulty, and practical scale.
- 3 BrowseComp-Plus: BrowseComp-Plus enables independent evaluation of retrieval and LLM components by extending BrowseComp with a document corpus.
- 3 BrowseComp-Plus: The corpus must provide complete reasoning-chain evidence, challenging distractors, and manageable computational size.
- 3 BrowseComp-Plus: The construction pipeline combines automated evidence mining, human verification, and hard-negative mining to produce a 100k-document corpus.
- 3.2.1 Evidence Document Gathering: OpenAI o3 retrieves candidate evidence pages and organizes them by clues, URLs, and supporting evidence.
- 3.2.1 Evidence Document Gathering: 124 BrowseComp pairs lacked o3 evidence, while 137 more were excluded because at least one cited URL could not be scraped, leaving 1,005 queries for verification.
- 3.2.2 Evidence Document Verification: Human verification checks clue-level justification and whether the evidence collectively supports answering the full question.
- 3.2.2 Evidence Document Verification: Annotators revise clues and search for additional evidence for at least 20 minutes when the initial documents fail verification.
- 3.2.2 Evidence Document Verification: Gold documents contain the final answer semantically, including cases where the answer is implicit rather than an exact substring.
4 Experiments
The experiments disentangle agent and retriever contributions through end-to-end, standalone retrieval, oracle, reasoning-effort, tool-use, and corpus-size analyses. Results show that stronger retrieval and greater reasoning effort generally improve accuracy, while open-source agents remain constrained in interleaved search-tool use.
- End-to-end performance: 70.12% accuracy is achieved by gpt-5 with Qwen3-Embedding-8B, while Qwen3-32B reaches 10.36% with the same retriever.Proprietary models substantially outperform the evaluated open-source models in end-to-end performance.
- End-to-end performance: More than 20 search calls per query are issued by gpt-5 and o3, compared with fewer than 2 from Qwen3-32B and SearchR1-32B.The results associate more exhaustive, multi-turn search with better outcomes on reasoning-intensive queries.
- Retriever effectiveness: Stronger retrieval improves final accuracy across all models, including gpt-5’s increase from 55.9% with BM25 to 70.12% with Qwen3-Embedding-8B.Qwen3-Embedding-8B and ReasonIR-8B also achieve higher retrieval recall and nDCG than BM25.
- Retriever effectiveness: Qwen3-Embedding-8B reduces search calls by approximately 1–3 for most proprietary models while improving effectiveness.The reported efficiency gains are attributed to higher precision in early search results and fewer follow-up queries.
- Oracle retrieval: 93.49% accuracy is reached by gpt-4.1 in the oracle setting, while Qwen3-32B reaches 83.25% when supplied with all labeled positive documents.The smaller oracle-setting gap indicates that sufficient evidence substantially narrows the difference between the two models, though Qwen3-32B also has context-window errors.
- Reasoning effort: 34.58% accuracy and 49.29% recall are reached by oss-20b with Qwen3-Embed-8B in high reasoning mode, versus 13.37% and 17.37% in low mode.Higher reasoning effort improves accuracy and recall but increases retrieval overhead; calibration error also tends to decrease.
5 Future Work and Discussion
BrowseComp-Plus is positioned as a controlled testbed for diagnosing agent–retriever interactions and supporting future research directions. The discussion highlights retriever quality, tool-use generalization, co-optimization, and commercial-search decomposition as open areas.
- Benchmark role: BrowseComp-Plus retains BrowseComp’s difficulty while enabling controlled, transparent analysis of deep-research systems.The benchmark is compared with earlier evaluation resources such as Natural Question and HotpotQA.
- Retriever optimization: Retriever quality affects agent learning dynamics, because BM25 may surface fewer relevant documents than embedding-based retrievers.The effect of retriever quality during agent optimization remains an open question.
- Tool-use generalization: Future work includes testing whether agents generalize when their search tool changes from BM25 to embedding-based retrieval.This targets out-of-distribution tool-use capabilities.
- Agent–retriever co-optimization: Future retrieval models could adapt to specific agents and be co-optimized for overall answer accuracy rather than evaluated in isolation.The paper frames this as a possibility because retrieved documents are consumed by tool-augmented LLM agents.
- Retrieval headroom: Oracle retrieval can greatly improve accuracy and may reduce search iterations, motivating high-precision retrieval for complex reasoning queries.BrowseComp-Plus provides a platform for fine-grained analysis of these interactions.
6 Conclusion
The conclusion presents BrowseComp-Plus as a reproducible benchmark for independently assessing retrieval and agent components. Experiments show that retrieval quality affects both answer effectiveness and search efficiency, while the benchmark supports future work on agent–retriever dynamics.
- Benchmark contribution: BrowseComp-Plus addresses reproducibility, fairness, and transparency by grounding queries in a fixed corpus with positive and hard-negative documents.This design enables independent and controlled assessment of retrieval and agent components.
- Empirical findings: Retrieval quality substantially impacts both the effectiveness and efficiency of deep-research systems.The conclusion bases this finding on experiments pairing diverse retrievers with open- and closed-source agents.
- Future directions: The benchmark supports future research on co-optimizing retrievers and agents, out-of-distribution tool-use generalization, and context engineering.The benchmark and baselines are described as publicly available.
B Labelling UI Example
Figure 5 shows a screenshot of the annotation interface used in the benchmark’s data-creation process.
- Interface: Figure 5 presents the annotation interface.The passage identifies the figure as a screenshot.
- Interface: The figure documents an interface used for annotation.Its role is presented as an example of the labeling workflow.
- Figure type: The visual is a screenshot rather than an evaluation result.No quantitative or comparative outcome is stated for the figure.
C Problematic Cases
The problematic cases section identifies flawed, ambiguous, or out-of-scope BrowseComp queries and describes decomposition into self-contained subqueries. These procedures support constructing a more suitable static document-based benchmark.
- Flawed question–answer pairs: Some BrowseComp question–answer pairs are flawed because people with the same name were incorrectly treated as the same individual.One example concerns a book author and a person who later returned to acting.
- Out-of-scope queries: 42 distance-related queries were removed because they require Google Maps access beyond a static document-based dataset.Static snapshots were not considered a valid substitute for the intended API capability.
- Ambiguous answers: 13 queries were removed because their expected answers were ambiguous or non-unique.The cited example had three credited usernames although the ground-truth answer listed only one.
- Query decomposition: Complex questions are decomposed into self-contained subqueries that preserve necessary context and constraints.The subqueries should independently target pieces of information needed for the original answer.
- Query decomposition: The example decomposition separates a multi-part fictional-character question into independently answerable character and television-show subqueries.The listed subqueries cover fourth-wall breaking, ascetic assistance, humor, airing period, and episode count.
E Main Search Prompt
The prompt instructs the agent to answer a question by interleaving reasoning with repeated search-tool interactions, then format the response into explanation, exact answer, and confidence sections.
- The agent must reason while interacting step by step with the provided search engine.The search tool may be used multiple times.
- The required response contains an explanation, an exact answer, and a confidence score.Evidence documents must be cited inline in the explanation using bracketed document IDs.
- The confidence score must be expressed between 0% and 100%.
F Evaluation Prompt
The evaluation prompt asks a judge to extract a response’s final answer, compare it with the supplied correct answer, explain meaningful differences, and assign correctness and confidence outputs.
- F Evaluation Prompt: The judge determines whether the response is correct against the supplied correct answer.
- F Evaluation Prompt: The judgment must follow the specified output format and criteria.
- F Evaluation Prompt: The extracted_final_answer records the response’s exact final answer, or ‘None’ when no exact final answer exists.
- F Evaluation Prompt: The judge receives the reference value through the [correct_answer] field.
- F Evaluation Prompt: The reasoning explains only meaningful differences between the extracted answer and the correct answer.It must not solve the original problem or argue for an alternative answer.
- F Evaluation Prompt: The correct field is ‘yes’ for a match or small numerical margin, and ‘no’ for inconsistency, ambiguity, non-equivalence, or error.
- F Evaluation Prompt: The confidence field extracts the response’s confidence percentage and defaults to 100 when unavailable.
- Table 8 is identified as presenting the overall API cost of proprietary agents.
H API Cost
The section identifies Table 8 as showing the API cost of the experiments.
- Table 8 shows the API cost of the experiments.
- The table is associated with the experiments’ cost reporting.
- The section concerns API-cost information presented in Table 8.