Source-linked AI summary
BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, Amelia Glaese
TL;DR
BrowseComp addresses the need for a difficult benchmark of web browsing beyond easily retrieved information. It uses human-created questions requiring persistent, strategic search while keeping answers short and verifiable. Results show that browsing alone yields limited gains, whereas Deep Research solves around half of the problems, and the benchmark remains an incomplete measure of general browsing ability.
Problem
Existing information-retrieval benchmarks largely emphasize easily found information and have become saturated by recent language models.
Method
BrowseComp uses 1,266 human-created questions requiring persistent browsing across many websites, with short answers that are easy to verify.
Results
Deep Research solves around half of BrowseComp problems, while enabling browsing for GPT-4o raises accuracy from 0.6% to 1.9%.
Takeaways & Limitations
BrowseComp measures core browsing capabilities involving persistence and creativity in finding hard-to-find information.
Takeaways & Limitations
BrowseComp does not guarantee that each question has no valid answer besides its provided reference answer.
Abstract
from arXiv · showhide
We present BrowseComp, a simple yet challenging benchmark for measuring the ability for agents to browse the web. BrowseComp comprises 1,266 questions that require persistently navigating the internet in search of hard-to-find, entangled information. Despite the difficulty of the questions, BrowseComp is simple and easy-to-use, as predicted answers are short and easily verifiable against reference answers. BrowseComp for browsing agents can be seen as analogous to how programming competitions are an incomplete but useful benchmark for coding agents. While BrowseComp sidesteps challenges of a true user query distribution, like generating long answers or resolving ambiguity, it measures the important core capability of exercising persistence and creativity in finding information. BrowseComp can be found at https://github.com/openai/simple-evals.
1 Introduction
BrowseComp targets browsing agents’ ability to find hard-to-locate information across many websites while keeping questions and answers short and easy to verify. It is designed as a challenging but limited benchmark of persistence, factual reasoning, and creative search.
- The benchmark addresses limitations of human browsing, including limited knowledge, distraction, fatigue, and inability to parallelize attention.
- BrowseComp introduces 1,266 challenging problems requiring browsing a large number of websites beyond simple information retrieval.
- Questions are designed to be challenging for people and existing models, while answers remain short, self-contained, and easy to verify against references.
- BrowseComp evaluates factual reasoning, persistent navigation, and creative searching needed to solve hard problems within a tractable time.
- The benchmark measures a limited subset of browsing capability and does not target long responses or resolving ambiguity in user queries.
2 Data collection and verification
BrowseComp was constructed from human-generated fact-seeking questions that are difficult to solve but simple to check. Its inverted-question design creates broad searches, while grading uses semantic equivalence to short reference answers.
- Data collection: Human trainers created challenging, fact-seeking questions with single, short answers intended to remain stable over time and be supported by evidence.
- Difficulty checks: Trainers checked difficulty by testing existing models, confirming answers were absent from initial search-result pages, and targeting more than ten minutes of human effort.
- Question design: The inverted-question process starts from a seed and combines characteristics with a large search space, making answers hard to find but easy to verify.
- Validity: BrowseComp questions are likely, but not guaranteed, to have only the provided valid answer because exhaustive checking for alternatives is impractical.
- Validity: Trainers were asked to add criteria when uncertain about uniqueness, and questions were revised when another trainer found a different valid answer quickly.
- Capabilities measured: The benchmark exercises factuality reasoning, persistent browsing, and creative search rather than brute-force enumeration alone.
3 Human performance on BrowseComp
Human trainers solved fewer than one-third of BrowseComp problems, and solved answers matched the reference answer less often than not perfectly. Solved questions varied widely in search time, while trainers who gave up had searched for about two hours.
- 29.2% of BrowseComp problems were solved by human trainers, with 86.4% of those solved answers matching the original reference answer.The evaluation covered 1,255 attempted examples out of 1,266 dataset questions.
- Trainers attempted questions without AI assistants and could give up only after up to two hours of searching.They were not given the correct answers and were instructed not to use ChatGPT, Claude, Perplexity, Grok, or Gemini.
- Solved questions ranged from less than one hour to two or three hours of searching.The time distribution was self-reported by trainers and is shown in Figure 3.
- Questions where trainers gave up were attempted for around two hours under the evaluation rules.Figure 3 compares the time distributions for solved problems and problems where humans gave up.
- The human results underscore BrowseComp's difficulty, though experienced professionals might solve some abandoned problems with ample time.The trainers were experienced with the dataset but were not competition-level internet browsers.
4 Evaluation of models
BrowseComp exposes a large gap between ordinary models and Deep Research, while showing that browsing, reasoning, persistent search, and additional test-time computation each matter. The evaluation also reveals calibration weaknesses, heterogeneous task difficulty, and the importance of strategic search beyond retrieval.
- Model performance: GPT-4o with browsing improves accuracy from 0.6% to 1.9%, but performance remains low.The comparison indicates that browsing alone is insufficient without strategic reasoning and interpretation of retrieved content.
- Model performance: Deep Research significantly outperforms all other evaluated models, solving around half of BrowseComp’s problems.Its autonomous web search, multi-source synthesis, and adaptive search strategy support difficult niche questions.
- Calibration: Models with browsing capabilities exhibit higher calibration error, suggesting web access can increase confidence in incorrect answers.Deep Research’s confidence scores are not well-calibrated in absolute terms, despite providing a meaningful internal signal for selecting correct answers.
- Test-time compute scaling: Performance scales smoothly with additional test-time browsing compute, with each point representing a full evaluation run at a different browsing effort.The scaling analysis motivates using more computation for questions requiring searches across many websites.
- Aggregation strategies: 15% to 25%: majority, weighted, and Best-of-N voting improve performance over a single attempt.Best-of-N consistently achieves the highest accuracy among the tested aggregation methods.
- Task difficulty: Deep Research solves 16% of tasks perfectly but fails entirely on 14%, while many tasks have intermediate pass rates.When given ground-truth answers, it usually retrieves supporting evidence for never-solved tasks, indicating that strategic perseverance and clue assembly are central challenges.
5 Related work and discussion
BrowseComp targets internet-enabled models that must persistently search for hard-to-find information, extending prior retrieval benchmarks focused on easily found facts. Its scope is deliberately limited, and future variants could cover additional modalities.
- Prior information-retrieval benchmarks mostly focused on facts humans could find easily, becoming saturated by recent language models.
- BrowseComp evaluates models with internet access on persistent search for hard-to-find information.
- The benchmark measures factuality, persistent navigation, and creative search, but not long-response ergonomics or resolving ambiguous user queries.
- Future benchmarks could test information-finding through images, video, audio, or interactive webpages rather than text alone.
A Additional instruction for model prediction
The additional instruction requires a structured response containing an explanation, an exact answer, and a confidence score.
- The model response must include an explanation, an exact answer, and a confidence score between 0% and 100%.
B Grading prompt
The grading prompt extracts a final answer and confidence from a response, then judges exact-answer equivalence against a precise reference answer.
- The grader evaluates a response against a precise and unambiguous correct answer.
- The grading output must follow a specified format and criteria.
- The grader extracts the final exact answer, using “None” when no exact final answer is available.
- The prompt records the reference answer and explains why the extracted answer is correct or incorrect.
- Answers are marked yes when they match the reference or fall within a small numerical error margin, and no otherwise.
- The grader extracts a confidence score from 0% to 100% and assigns 100 when no score is available.