PaperlayerView exact revision on arXiv

Source-linked AI summary

MedBrowseComp: Benchmarking Medical Deep Research and Computer Use

Shan Chen, Pedro Moreira, Yuxin Xiao, Sam Schmidgall, Jeremy Warner, Hugo Aerts, Thomas Hartvigsen, Jack Gallifant, Danielle S. Bitterman

arXiv:2505.14963v1cs.CL

TL;DR

Static medical QA benchmarks are near saturation and do not reflect the live-web challenges of reconciling current evidence across sources. MedBrowseComp tests agents on multi-hop medical fact retrieval using a physician-informed, dynamically updateable benchmark. No system exceeds 40% accuracy on questions requiring more than one hop.

  • Problem

    Static medical benchmarks are near saturation and often avoid live-web hurdles, while clinicians must reconcile current evidence scattered across heterogeneous sources.

    Source: §1 · p. 2

  • Method

    MedBrowseComp evaluates web-browsing agents on short, verifiable medical fact questions built with physicians using HemOnc.org.

    Source: §1 · p. 2

  • Results

    40% is the accuracy ceiling for every system on questions requiring more than a single hop, and GUI-centric agents underperform API-driven deep researchers.

    Source: §5.2 · p. 9

  • Takeaways & Limitations

    MedBrowseComp provides a realistic, challenging test bed for deep-research systems and a demanding but attainable subset for computer-use agents.

    Source: §5.2 · p. 9

    Querying multiple distinct agents for every question is computationally expensive, underscoring the need for more efficient unified systems.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

Large language models (LLMs) are increasingly envisioned as decision-support tools in clinical practice, yet safe clinical reasoning demands integrating heterogeneous knowledge bases -- trials, primary studies, regulatory documents, and cost data -- under strict accuracy constraints. Existing evaluations often rely on synthetic prompts, reduce the task to single-hop factoid queries, or conflate reasoning with open-ended generation, leaving their real-world utility unclear. To close this gap, we present MedBrowseComp, the first benchmark that systematically tests an agent's ability to reliably retrieve and synthesize multi-hop medical facts from live, domain-specific knowledge bases. MedBrowseComp contains more than 1,000 human-curated questions that mirror clinical scenarios where practitioners must reconcile fragmented or conflicting information to reach an up-to-date conclusion. Applying MedBrowseComp to frontier agentic systems reveals performance shortfalls as low as ten percent, exposing a critical gap between current LLM capabilities and the rigor demanded in clinical settings. MedBrowseComp therefore offers a clear testbed for reliable medical information seeking and sets concrete goals for future model and toolchain upgrades. You can visit our project page at: https://moreirap12.github.io/mbc-browse-app/

1 Introduction

Medical information seeking requires reconciling current evidence scattered across heterogeneous sources, but existing benchmarks do not systematically test agents’ ability to do this through live-web research. MedBrowseComp evaluates multi-hop medical fact retrieval and reports that frontier systems remain below 50% overall, with accuracy below 10% on its two hardest question sets.

1 Introduction

  • Medical research and clinical decisions require integrating current findings, guidelines, and safety advisories scattered across heterogeneous sites.Existing evaluations lack a unified benchmark for systematic, large-scale assessment of complex multi-source medical retrieval.Source: §1 · p. 2
  • Static medical benchmarks are near saturation and often sidestep live-web hurdles such as obsolete links, contradictory evidence, and shifting page layouts.The paper argues that medical agents need to be tested on multi-hop searches that navigate, filter, and reconcile real-world information.Source: §1 · p. 2
  • MedBrowseComp evaluates agents’ ability to browse general and specialized medical sources to locate verifiable facts.It was designed collaboratively with physicians using HemOnc.org, a wiki maintained weekly by oncologists for six years.Source: §1 · p. 2
  • Frontier deep-research systems and computer-use agents score below 50% overall and below 10% on the two hardest question sets.The benchmark uses short, objective, verifiable fact-seeking questions designed to simplify evaluation.Source: §1 · p. 2

2 Related Work

Prior benchmarks assess general tool use, web traversal, retrieval quality, factual correctness, or specialized knowledge, but do not combine those demands for medical information seeking. MedBrowseComp focuses on multi-hop searches across reputable medical sources and is designed to track evolving online knowledge.

2 Related Work

  • GAIA tests multimodal input, open-ended reasoning, and external-tool use, while WebWalker and FRAMES emphasize web traversal and retrieval-augmented generation.SimpleQA tests short-answer factual correctness but remains relatively easy to solve with basic web searches.Source: §2 · p. 3
  • BrowseComp targets difficult-to-find facts, while Humanity’s Last Exam emphasizes specialized knowledge and reasoning more than source navigation.BrowseComp-ZH extends difficult fact finding to the Chinese web and its regional constraints.Source: §2 · p. 3
  • MedBrowseComp’s 1,000+ questions require exploration of reputable medical sources, interpretation of terminology, and alignment with evidence-based standards.Its expandable design allows updates as medical knowledge and online resources evolve.Source: §2 · p. 4

3 Methods

The benchmark is built from physician-curated hematology and oncology data linked to publications, trials, drug approvals, and market data. The evaluation spans API-accessible research systems and a computer-use agent, with a restricted task subset for GUI evaluation.

3 Methods

  • HemOnc.org supplies physician-verified hematology and oncology content, including over 1,000 pages, 5,455 treatment regimens, and 6,950 referenced clinical studies.The resource uses a standardized ontology and is available through the HemOnc Dataverse.Source: §3 · pp. 4–5
  • The authors cleaned regimen efficacy data, linked cases to PubMed publications and ClinicalTrials.gov information, and collected data through April 2025.They excluded trials linked to multiple publications, matched regimen drugs to FDA Orange Book data, and manually verified and deduplicated a final set of 121 trials.Source: §3 · p. 5

3.1 Model Selection and other Details

  • Systems with accessible APIs and costs were evaluated on MedBrowseComp-605, while systems without those features used the smaller MedBrowseComp-50 set.Two annotators achieved 100% agreement with each other and 98% agreement with the GPT-4.1 mini judge on MedBrowseComp-50.Source: §3.1 · p. 5
  • The study selected Anthropic’s Computer Use agent with Claude 3.7 Sonnet because its documented API and sandbox supported large-scale automated runs.The GUI benchmark was restricted to trial IDs, second authors, PMIDs, and start dates retrievable within one or two browser actions.Source: §3.1 · p. 5

4 Results

Deep-research accuracy improves with iterative browsing but declines as hop count rises, while retrieval augmentation offers diminishing gains on deeper MedBrowseComp-605 questions. Computer-use performance varies by extraction task, with plausible-paper confusion weakening PMID retrieval.

4.1 Deep Research Agent Results

  • 25.5/50 O3 deepresearch answers were correct versus 19/50 O3 search (34% relative gain); Gemini-2.5-pro deepsearch improved 75% (24.5 vs. 14), with strongest gains at 4- and 5-hop splits.Figure 3 summarizes MedBrowseComp-50 accuracy across hop counts.Source: §4.1 · p. 6
  • Accuracy declined monotonically with hop count across systems, while deep-research agents more than doubled baseline accuracy on the hardest 4- and 5-hop splits.The text identifies long-horizon web navigation as an ongoing challenge for frontier agents.Source: §4.1 · p. 6
  • On MedBrowseComp-605, retrieval-augmented generation improved overall performance, but its benefit diminished with increasing hop count and was virtually absent at 4 and 5 hops.Bare parameter-only systems struggled across most tasks; retrieval gains were substantial on shallow questions.Source: §4.1 · pp. 6–7
  • O3 deepresearch and Gemini-2.5-pro deepsearch led the evaluated systems, behind the post-hoc Best of 1 upper bound (30/50); Sonar Pro scored 10/50 and Perplexity deepresearch 20/50.The authors say the comparison may suggest instruction-tuned LLMs can outperform specialized retrieval systems when granted autonomous browsing.Source: §4.1 · p. 7
  • Cross-model test-time compute improved overall accuracy beyond repeated sampling from one system, but querying multiple agents for each question was substantially costly.The authors call for more efficient unified systems that deliver comparable or better results with lower resource overhead.Source: §4.1 · p. 7

4.2 Computer Use Agent Results

  • The computer-use diagnostic subset covers four HemOnc tasks—Clinical Trial IDs, Start Dates, Second Authors, and PubMed IDs—and requires no more than two browser actions.It is designed to isolate retrieval and extraction of specific biomedical fields with minimal navigational complexity.Source: Table 2 · p. 8, §4.2 · p. 8
  • The agent performed best on Second Author and Trial ID extraction; Start Date errors often came from choosing the first date rather than the dedicated Study Start field.Second Author and Trial ID information is often nearby on the same page, while Start Date tasks require navigating to ClinicalTrials.gov.Source: §4.2 · p. 8
  • Matching a PMID was the weakest task: the agent often returned a valid PMID for a plausible but incorrect paper, indicating semantic confusion rather than visual or interface failure.The authors distinguish this error pattern from common visual brittleness or interface failures.Source: §4.2 · p. 8

5 Conclusion and Future Work

MedBrowseComp exposes persistent difficulty on multi-hop questions and a performance gap between deep researchers and GUI agents. The authors note limits in system coverage, evaluation, cost, and clinical scope, and propose broader tasks and real-world studies.

5 Conclusion and Future Work

  • Evaluation covered only agents with publicly programmable APIs, leaving systems such as Operator and Mariner unevaluated.The authors say it is unclear whether including them would substantially change the leaderboard, given their performance relative to Claude CUA on other benchmarks.Source: §5 · p. 9
  • All answers were machine-judged with a lightly audited LLM rubric, although a subset was human-verified with good agreement.Source: §5 · p. 9
  • Building the benchmark and running baselines cost $3,690 in total.The listed costs include about $2,500 for Claude Sonnet 3.7 CUA.Source: §5 · p. 9
  • The study did not present answers to clinicians, and its questions cover only a small portion of the knowledge used in real-world clinical decision-making.The authors note that expert-curated, verifiable knowledge is not available for the entire medical domain.Source: §5 · p. 9

5.1 Future Work

  • Future work proposes broader tasks, tool adapters for computer-use agents, and benchmarking closed systems under identical prompts.Source: §5.1 · p. 9
  • Future studies should test agents in AI-IDE environments and compare agentic systems with humans and in real clinical settings.Source: §5.1 · p. 9

5.2 Conclusion

  • No system exceeds 40% accuracy on questions requiring more than one hop, and retrieval-augmented text pipelines answer nearly half as many questions as deep researchers.GUI-centric agents also tend to underperform API-driven deep researchers; their workflow uses roughly 200,000 tokens versus roughly 2,000 for an equivalent text-only workflow.Source: §5.2 · p. 9
  • GUI-centric agents tend to underperform API-driven deep researchers, while their screenshot loop uses roughly 200,000 tokens versus roughly 2,000 for an equivalent text-only workflow.Deep-research pipelines reach structured endpoints in a single call, reducing latency and cost.Source: §5.2 · p. 9

Contribution

The contribution record names the authors’ roles in framework development, experiments, writing, data collection, curation, supervision, and project oversight.

Contribution

  • Shan Chen and Pedro Moreira contributed equally, developing the main theoretical framework, performing experiments, and leading manuscript writing.Source: Contribution · p. 13
  • Yuxin Xiao contributed to experimental design and data collection, while Sam Schmidgall, Jeremy Warner, and Hugo Aerts contributed to dataset curation.Source: Contribution · p. 13
  • Thomas Hartvigsen and Jack Gallifant provided supervision and strategic direction; Danielle S. Bitterman oversaw the project, coordinated the team, secured funding, mentored junior members, and approved the final version.Source: Contribution · p. 13

A.1 Detailed Benchmark Setting

The passage identifies Table 4 as an overview of test runs for the MedBrowseComp benchmarks.

A.1 Detailed Benchmark Setting

  • Table 4 provides an overview of test runs for the MedBrowseComp benchmarks.Source: Table 4 · p. 14

A.2 LLM as judge Prompts

The authors describe a modified judging prompt and explain that they omitted confidence estimation because API access was unavailable for many models.

A.2 LLM as judge Prompts

  • The judging instruction was adapted from Humanity’s Last Exam and used an OpenAI structured-output pipeline with a JSON-only system prompt.Source: §A.2 · p. 15
  • Confidence estimation was omitted because the authors lacked API access to many models.Source: §A.2 · p. 15

Pydantic Schema for Structured Output

The schema extracts an exact final answer from a response and constrains correctness reasoning to comparing that answer with the supplied correct answer.

Pydantic Schema for Structured Output

Grading Prompt

The grading prompt extracts a precise final answer, compares it with the supplied correct answer, and judges whether they match.

Grading Prompt

  • The prompt extracts the response’s final exact answer, using “None” if no exact answer is present.It directs the judge to focus on the extracted answer rather than solving the question anew.Source: Grading Prompt · p. 15
  • The judge explains correctness by checking for meaningful differences between the extracted answer and the supplied correct answer.The prompt excludes background commentary and arguments for answers other than the supplied one.Source: Grading Prompt · p. 15
  • The judge returns “yes” for a match or a small numerical margin of error, and “no” for inconsistency, ambiguity, or an incorrect answer.The prompt treats non-equivalence as incorrect.Source: Grading Prompt · p. 15

A.3 Computer Use Agent Prompt

The appendix documents the computer-use prompt and reports findings on medical question answering and GUI-agent extraction, alongside a checklist covering study disclosures and asset documentation guidance.

A.3 Computer Use Agent Prompt

  • The computer-use prompt largely follows Anthropic’s version, adding an instruction for the agent to act independently rather than request clarification or human help.Its operational guidance covers Ubuntu and Firefox use, browsing and scrolling, and downloading PDFs for text extraction when needed.Source: §A.3 · p. 16

A.4 Deep Research Agent Results

  • 24.5% was Gemini 2.5 Pro (search)’s REAL accuracy on MedBrowseComp 50, the highest reported under conditions excluding “Not applicable” answers.The authors describe the benchmark as requiring multi-step reasoning across medical knowledge bases.Source: §A.4 · p. 17, Table 5 · p. 17
  • 76% on 1-hop questions fell to 5.1% REAL accuracy on 4-hop questions for Gemini 2.5 Pro, with 5-hop performance described as essentially zero.The authors argue that this illustrates the difficulty of chaining multiple reasoning steps and that complex medical reasoning remains far from trustworthy without supervision.Source: §A.4 · p. 17

A.5 Evaluation from Optimal Start Page

  • 42.98% versus 11.57% was Claude 3.7’s PMID extraction accuracy with versus without initialization from HemOnc.org, respectively.The authors report that scores improved notably compared with Table 4, particularly for PMID extraction.Source: §A.5 · p. 19, Table 8 · p. 19
  • HemOnc.org’s structured, domain-specific navigation may reduce ambiguity and detours, shorten tasks to one or two clicks, and align agent workflows with clinicians’ use of the resource.The authors suggest formal initialization policies as a future component of web-agent pipelines.Source: §A.5 · p. 19

NeurIPS Paper Checklist

  • The checklist says the authors provide public code and repository README instructions, along with downloadable counts, outputs, and LLM-as-judge results.It also states that no training or tuning was conducted and that code, settings, and outputs were provided.Source: NeurIPS Paper Checklist · pp. 21–22
  • The authors say they discuss societal impacts in an ethics section and release an encoded dataset to reduce pretraining-contamination risk from GitHub scraping.They state that the models and datasets used are already publicly available.Source: NeurIPS Paper Checklist · pp. 23–24
  • Asset guidance calls for citing the original work and identifying each asset’s version and, where possible, its URL.Source: NeurIPS Paper Checklist · p. 24
  • Asset guidance also calls for license, copyright, and terms-of-service information, including for scraped or repackaged datasets, and encourages contacting creators when details are unavailable.Source: NeurIPS Paper Checklist · p. 25
  • The authors say dataset, code, and findings documentation is available on their website, alongside a more accessible blog explaining the approach and findings.Source: NeurIPS Paper Checklist · p. 25
  • The paper reports no crowdsourcing and says the research team gathered and validated the results.Source: NeurIPS Paper Checklist · p. 25
  • The authors state that the paper involves no human subjects and therefore marks the IRB-approval question not applicable.Source: NeurIPS Paper Checklist · p. 25
  • The checklist marks LLM-use disclosure not applicable because LLMs are not used as the core methodology.Source: NeurIPS Paper Checklist · p. 26

Related papers

Loading 2505.14963v1…