PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery

Lei Xiong, Kun Luo, Ziyi Xia, Wenbo Zhang, Jin-Ge Yao, Zheng Liu, Jingying Shao, Jianlyu Chen, Hongjin Qian, Xi Yang, Qian Yu, Hao Li, Chen Yue, Xiaan Du, Yuyang Wang, Yesheng Liu, Haiyu Xu, Zhicheng Dou

arXiv:2604.25256v1cs.AI

TL;DR

Existing web-browsing benchmarks do not measure whether agents can deeply search, verify, and complete scientific literature discovery. AutoResearchBench evaluates this through Deep and Wide Research tasks, finding frontier models achieve only 9.39% accuracy and 9.31% IoU, respectively.

  • Problem

    Existing web-browsing benchmarks leave largely unmeasured whether agents can search scientific corpora, verify fine-grained technical conditions, and determine when literature searches are complete.

  • Method

    AutoResearchBench uses Deep Research for precise target-paper identification and Wide Research for comprehensive collection of papers satisfying scientific specifications.

  • Results

    Frontier models achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research, while many strong baselines fall below 5%.

  • Takeaways & Limitations

    AutoResearchBench shows that current frontier agents remain far from saturation on rigorous autonomous scientific literature discovery.

  • Takeaways & Limitations

    The benchmark currently focuses on computer science papers in a fixed corpus and mainly evaluates text-based search and reasoning.

Abstract

from arXiv · show

Autonomous scientific research is significantly advanced thanks to the development of AI agents. One key step in this process is finding the right scientific literature, whether to explore existing knowledge for a research problem, or to acquire evidence for verifying assumptions and supporting claims. To assess AI agents' capability in driving this process, we present AutoResearchBench, a dedicated benchmark for autonomous scientific literature discovery. AutoResearchBench consists of two complementary task types: (1) Deep Research, which requires tracking down a specific target paper through a progressive, multi-step probing process, and (2) Wide Research, which requires comprehensively collecting a set of papers satisfying given conditions. Compared to previous benchmarks on agentic web browsing, AutoResearchBench is distinguished along three dimensions: it is research-oriented, calling for in-depth comprehension of scientific concepts; literature-focused, demanding fine-grained utilization of detailed information; and open-ended, involving an unknown number of qualified papers and thus requiring deliberate reasoning and search throughout. These properties make AutoResearchBench uniquely suited for evaluating autonomous research capabilities, and extraordinarily challenging. Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research, while many other strong baselines fall below 5%. We publicly release the dataset and evaluation pipeline to facilitate future research in this direction. We publicly release the dataset, evaluation pipeline, and code at https://github.com/CherYou/AutoResearchBench.

1 Introduction

AutoResearchBench evaluates autonomous scientific literature discovery through precise target-paper identification and comprehensive literature collection. Its full-text, open-ended tasks reveal that even frontier agents struggle substantially, achieving below 10% on both task types.

  • Benchmark design: AutoResearchBench contains 1,000 expert-verified problems across eight computer-science areas, built through a full-text-first human–machine pipeline.The benchmark includes Deep Research and Wide Research task families.
  • Task types: Deep Research tracks down a specific target paper through progressive multi-step probing, whereas Wide Research comprehensively collects papers satisfying given conditions.Deep Research can also require concluding that no paper satisfies the condition.
  • Task characteristics: The benchmark is research-oriented, literature-focused, and open-ended, requiring technical understanding, fine-grained full-paper evidence, and careful decisions about exploration, verification, abstention, and stopping.Relevant clues may come from tables, figures, and reference lists, while the number of qualifying papers can be zero or otherwise unknown.
  • Evaluation infrastructure: The evaluation infrastructure operates over more than three million up-to-date arXiv papers with extracted full-text content and search tools.The authors publicly release the dataset and evaluation pipeline to support controlled, large-scale evaluation and future research.
  • Evaluation results: 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research are achieved by frontier models, while many strong baselines fall below 5%.These results indicate a substantial gap between scientific literature search and performance on general web-browsing benchmarks.

2 AutoResearchBench

AutoResearchBench evaluates autonomous scientific literature discovery through complementary Deep Research and Wide Research tasks. Its construction combines full-text scientific evidence, corpus-scale search, model assistance, and human verification to test precise identification and exhaustive coverage.

  • Benchmark formulation: AutoResearchBench evaluates agents that search, browse, and reason over scientific papers to produce an answer set from a query and corpus.At each step, the agent maintains the query, interaction history, and observed documents while using search tools or reasoning.
  • Academic Deep Research: Deep Research requires identifying the unique paper satisfying tightly coupled, obfuscated full-text and citation-based constraints, or establishing that no such paper exists.Its ground-truth set contains either zero or one paper, and each clue is weak in isolation while their conjunction identifies the target.
  • Construction pipeline: Benchmark construction uses a curated scientific corpus and combines model-assisted generation with human verification for realistic, challenging literature-discovery tasks.The design targets long-horizon exploration, full-text inspection, and scientific reasoning.
  • Deep Research construction: Deep Research instances are built by mining and fuzzifying full-text constraints, pruning them until they uniquely locate a target, and verifying answerability or no-answer status.The pipeline avoids headline cues, incorporates citation-based multi-hop clues, and tests instances with frontier LLM agents and human annotators.
  • Academic Wide Research: Wide Research requires exhaustively retrieving all papers satisfying a scientific query while preserving semantic precision across a large corpus.The task emphasizes recall-oriented exploration, evidence-based filtering, and tracing the boundary of the target concept.
  • Wide Research construction: Wide Research construction uses academic entity graphs to source candidates, abstract shared attributes, formulate constrained queries, and verify an initial approximation of the complete valid set.Human annotators preserve logical constraints and add missing valid papers during initial verification.

3 Experiments

Experiments show that autonomous scientific literature search remains largely unsolved: leading systems stay below 10% on both benchmark tasks. Failures reflect weak evidence-seeking, scientific reasoning, comprehensiveness, and iterative search, while tool choice and test-time scaling materially affect outcomes.

  • Overall Performance: 9.39% accuracy is Claude-Opus-4.6’s peak on Deep Research, while Gemini-3.1-Pro-Preview reaches only 9.31% IoU on Wide Research.The majority of evaluated models, including open-source systems, score below 5%.
  • Overall Performance: 7.44% accuracy is achieved by GPT-5.4 in Deep Research with 6.1 turns, whereas longer trajectories can produce lower accuracies.Kimi-K2.5 uses 28.8 turns and achieves 4.2% accuracy; redundant searches and invalid reasoning often fail to acquire discriminative evidence.
  • Failure Analysis: Scientific reasoning over complex constraints is the primary bottleneck, with Claude-Opus-4.6 averaging 28.1 turns yet failing in over 90% of Deep Research cases.Agents often identify plausible candidates but struggle to isolate a unique target when evidence is obfuscated or embedded deep in full texts.
  • Failure Analysis: Comprehensive Wide Research requires long natural-language queries, systematic hypothesis-space management, and complete result sets, capabilities in which agents perform poorly.Short-query preferences optimized for general web tasks degrade performance in scientific scenarios.
  • Search Tools: 5.42% Deep Search accuracy with DeepXiv falls to 3.97% with open-web search, averaged over four matched models.Full-text-accessible academic indexing better matches queries built from paper-internal evidence that web-visible summaries often omit.
  • Scaling and Reasoning: Test-time scaling consistently helps Deep Research more than Wide Research, while THINK generally harms Wide Research despite increasing runtime.Repeated runs address brittle trajectory decisions more effectively than recall bottlenecks, where similar omissions recur.

4 Related Work

Prior academic search agents advance deep, long-horizon web search through iterative planning and multi-step reasoning, but existing benchmarks remain inadequate for rigorous, evidence-based scientific literature evaluation.

  • Academic Search Agents: Academic search agents increasingly target complex, long-horizon information search through deep search and deep research paradigms.These agents demonstrate iterative planning, multi-step reasoning, and web navigation capabilities.
  • Academic Search Agents: Scientific research remains more demanding than open-domain web search because it requires processing peer-reviewed literature and understanding scientific content.
  • Benchmark of Academic Search Agents: Existing agent benchmarks mainly evaluate general-domain capabilities, while other datasets isolate specific search patterns or combine deep and wide paradigms.Examples include GAIA and BrowseComp for broad autonomous capabilities; WideSearch and InfoDeepSeek for specific search patterns; and DeepWideSearch, GISA, and Table-as-Search for unified paradigms.
  • Benchmark of Academic Search Agents: Because open-web datasets lack controlled corpora, they are unsuitable for rigorous, evidence-based evaluation of academic search agents.

5 Conclusion · A Ethic Statement

AutoResearchBench evaluates autonomous agents on rigorous scientific literature investigations using expert-curated queries and a large, contamination-resistant full-text corpus. Its ethical framing emphasizes public scientific sources, document-grounded verification, and support for reliable search without replacing expert judgment.

  • 5 Conclusion: AutoResearchBench systematically evaluates autonomous agents on rigorous scientific literature investigations.The benchmark targets autonomous research performance in scientific literature discovery.
  • 5 Conclusion: 1,000 expert-curated queries form the benchmark’s evaluation set.These queries are grounded in a controlled corpus containing over 3 million full-text papers.
  • 5 Conclusion: The benchmark uses a controlled, contamination-resistant corpus of over 3 million full-text papers.Its corpus supports systematic evaluation of scientific literature investigations.
  • 5 Conclusion: AutoResearchBench tests the intersection of long-horizon document browsing and complex scientific reasoning.The evaluation spans the Deep Research and Wide Research paradigms.
  • A Ethic Statement: The benchmark is constructed from publicly accessible scientific papers in arXiv/DeepXiv.It does not involve private personal data or direct interaction with human subjects.
  • A Ethic Statement: Human annotators and model assistance are used only for query construction and verification.Final decisions are grounded in document evidence.
  • A Ethic Statement: AutoResearchBench is intended to improve reliable scientific search rather than replace expert judgment.The ethics statement frames the benchmark as an evaluation resource for reliable scientific search.

B Limitations and Future Works

AutoResearchBench currently centers on computer science papers in a fixed corpus and evaluates primarily text-based search and reasoning. Future work should extend coverage to cross-domain science, multimodal evidence, continually evolving literature, and more exhaustive wide-search answers.

  • Scope limitations: The benchmark focuses on computer science papers within a fixed corpus, limiting coverage of other scientific domains.Cross-domain science remains a direction for future study.
  • Scope limitations: Its evaluation mainly targets text-based search and reasoning, leaving multimodal evidence for future study.The current design does not cover evidence beyond text.
  • Future work: Continually evolving literature is another area not covered by the current benchmark.Future versions could address literature that changes over time.
  • Wide-search limitations: Even with rigorous verification, exhaustive answer sets for wide search remain challenging at the boundary of large collections.The passage identifies this as an additional limitation of wide-search evaluation.

C Detailed Statistic of AutoResearchBench … E.1.3 Tool Implementation

AutoResearchBench spans eight computer-science research areas, uses deliberately difficult answer distributions, and relies on extensive multi-stage verification and resource-intensive evaluation. Its experimental implementation combines ReAct agents, asynchronous execution, configurable runtimes, and unified paper/web search tooling.

  • C Detailed Statistic of AutoResearchBench: Deep and Wide Research span all eight major computer-science research areas despite differing topical proportions.This provides broad coverage of key subfields within computer science.
  • C Detailed Statistic of AutoResearchBench: Approximately 10% of Deep Research queries have no correct answer in the three-million-paper corpus, while Wide Research answers have a long-tail distribution.Most Wide Research queries contain fewer than ten valid answers, while popular areas may yield more than twenty candidates.
  • D.1 Deep Research Verification Pipeline: Deep Research instances survive four adversarial stages: shortcut screening, agent stress testing, timed human retrieval, and uniqueness auditing.Instances are retained only after passing all four stages, including a fixed 10-minute human search test and corpus-level uniqueness audit.
  • D.2 Wide Research Verification Pipeline: Wide Research verification expands candidates through approximately ten diverse queries per question, yielding 31,734 candidate papers before screening.The ensemble uses Gemini-3.1-pro, Claude 4.6-Sonnet, and GPT-5.4 with Jina search, followed by abstract filtering and temporal normalization.
  • Stage 3: Hybrid Machine-Human Verification: 4,887 papers (24.1%) passed majority voting after 20,251 (87.2%) screened candidates were retrieved for full-text analysis.The resulting 75.9% rejection rate reflects strict multi-condition filtering, with human audits triggered when compliance falls below 75% precision.
  • E.1.1 Evaluation Framework: Evaluation uses a single-tool ReAct loop that distinguishes wide from deep research and supports multiple trajectories for pass@k metrics.Runs are asynchronous, resumable through line-oriented JSON records, and terminate on finish markers, context or turn limits, or API failures.
  • E.1.2 Model and Runtime Hyperparameters: The planner uses moderate sampling, long HTTP timeouts, retries, fixed per-turn completion caps, and optional extended reasoning, while tool summarizers disable extended reasoning.Tool-side summarization uses deterministic JSON-style outputs with thinking disabled.
  • E.1.3 Tool Implementation: Paper and web search backends normalize hits into shared agent-facing results with query-conditioned snippets or partial text compressed into evidence strings.The interface accepts a required query and optional top_k=10, dispatches to the configured backend, and injects JSON results inside <tool_response> tags.

E.2 Detailed Results and Analysis

The section reports Deep Research and Wide Research results with task-specific effectiveness and efficiency metrics. A manual audit finds that 96% of extra Wide Research predictions were valid true negatives, supporting dataset comprehensiveness and attributing low precision mainly to model-side reasoning errors.

  • Deep Research: Deep Research measures exact-match accuracy for pinpointing a singular target document through multi-hop citation graphs.It also reports average maximum turns, temporal cost, token consumption, and tool invocation frequency to characterize exploration overhead.
  • Wide Research: Wide Research evaluates constrained set retrieval using IoU, Precision, and Recall alongside Time, Tokens, and Turns.Decomposing IoU into Precision and Recall exposes models that retrieve broadly but violate strict query constraints.
  • Validity Audit: 96% of extra predictions from three top-performing models were valid true negatives in blind human expert review.The audited papers violated at least one explicit constraint, including evaluation setting, methodology, or temporal requirements.
  • Validity Audit: The audit supports the comprehensiveness of the dataset construction pipeline and attributes low precision primarily to model-side reasoning errors.The audit addresses the threat that incomplete ground truth could penalize valid but unannotated papers.

F Detailed Error Analysis

The paper analyzes systematic failure modes by manually inspecting incorrect predictions from three frontier agents across Deep Research and Wide Research. It uses mutually exclusive error taxonomies and reports each category’s proportion per model, with uncategorized residual errors grouped as O.

  • Analysis setup: The analysis manually inspects incorrect predictions from Gemini-3.1-Pro, Seed-2.0-Pro, and Claude Opus 4.5.The sample is stratified across both task tracks.
  • Analysis setup: The study covers both Deep Research and Wide Research tasks.Error types are defined separately for each track.
  • Error taxonomy: Each track uses a mutually exclusive error taxonomy to measure the proportion of each error type per model.Figure 11 visualizes the resulting error distributions.
  • Error taxonomy: A small fraction of errors that do not fit any category cleanly is grouped as O.This residual category captures errors outside the defined taxonomy.

F.1 Error Types · F.1.1 Deep Research Error Analysis

The paper’s Deep Research error taxonomy identifies failures in retrieval, execution, evidence aggregation, and candidate ranking. These errors show that success requires full-constraint verification, robust tool use, stateful evidence tracking, and accurate final reranking.

  • F.1.1 Deep Research Error Analysis: Local semantic drift is dominant: agents reach the correct topical neighborhood but select nearby papers matching only salient clues.Common confusions include a cited anchor paper versus the target or closely related subfields; the core issue is incomplete full-constraint verification, not recall alone.
  • F.1.1 Deep Research Error Analysis: Malformed tool calls, parser failures, and interrupted execution consume budget, disrupt evidence accumulation, and block targeted follow-up on unresolved constraints.When execution becomes unstable, agents often revert to generic hypotheses instead of recovering through disciplined verification.
  • F.1.1 Deep Research Error Analysis: All six trajectories ended with an empty candidate list despite averaging 22.7 turns and 119.2 retrieved papers.The failure reflects weak stateful reasoning over accumulated evidence rather than a lack of retrieved clues.
  • F.1.1 Deep Research Error Analysis: Agents often fail to track satisfied constraints, unresolved constraints, or which candidate best fits their conjunction of evidence.This evidence-aggregation problem also produces termination failures after retrieving many relevant clues.
  • F.1 Error Types: The error taxonomy covers DeepSearch and WideSearch, organizing failures into retrieval, execution, evidence aggregation, termination, candidate selection, and ranking.Table 11 presents the taxonomy for both benchmark tasks.
  • F.1.1 Deep Research Error Analysis: Candidate-ranking failures occur after successful retrieval, when the gold paper is already present but the top-ranked prediction is incorrect.These cases make exact reranking and consistency between intermediate reasoning and final selection first-order bottlenecks.

F.1.2 Wide Research Error Analysis · F.2 Case Study · G Use of LLM

Wide Research failures center on semantic boundary mismatches, unconstrained candidate expansion, rigid constraint handling, and domain-knowledge gaps. The paper also presents successful DeepSearch trajectories and documents LLM use in dataset construction and manuscript refinement.

  • F.1.2 Wide Research Error Analysis: 68.0% of Gemini-3.1-Pro errors involve GT semantic boundary misalignment, compared with 36.0% for Seed-2.0-Pro.The predicted answer is semantically adjacent to, but does not exactly match, the intended ground-truth entity.
  • F.1.2 Wide Research Error Analysis: 85.3% of Claude Opus 4.5 errors arise from precision-unconstrained candidate expansion, maximizing recall while neglecting specified constraints.This profile differs sharply from the model’s ranking-failure profile in DeepSearch.
  • F.1.2 Wide Research Error Analysis: Constraint literalism and premature termination cause agents to reject valid candidates early and stop without broadening their search strategy.Seed-2.0-Pro is inflexible during constraint application and insufficiently persistent during exploration.
  • F.1.2 Wide Research Error Analysis: 13.3% of Gemini-3.1-Pro errors are scientific knowledge coverage gaps caused by failing to interpret domain-specific terminology despite retrieving the candidate.This reflects limitations in domain-grounded understanding rather than search capability.
  • F.2 Case Study: The case study presents partial DeepSearch and WideSearch trajectories, including a correct DeepSearch trajectory demonstrating strong reasoning, self-reflection, and verification.Figures 14 and 15 illustrate the correct-solving trajectory.
  • G Use of LLM: During dataset construction, the authors employ LLMs to assist with data creation, while manuscript writing uses them for translation and linguistic refinement.The paper states these uses explicitly and refers to Section 2 for dataset-construction details.
  • G Use of LLM: An Opus trajectory iteratively broadens a computational-matching search from stereo matching to optical flow before investigating LiteFlowNet candidates.The trajectory includes query reformulation and candidate checking against architectural and author-affiliation constraints.
Loading 2604.25256v1…