PaperlayerView exact revision on arXiv

Source-linked AI summary

AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

Jonathan Bragg, Mike D'Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi, Sergey Feldman, Dany Haddad, Jena D. Hwang, Peter Jansen, Varsha Kishore, Bodhisattwa Prasad Majumder, Aakanksha Naik, Sigal Rahamimov, Kyle Richardson, Amanpreet Singh, Harshit Surana, Aryeh Tiktinsky, Rosni Vasu, Guy Wiener, Chloe Anastasiades, Stefan Candra, Jason Dunkelberger, Dan Emery, Rob Evans, Malachi Hamada, Regan Huff, Rodney Kinney, Matt Latzke, Jaron Lochner, Ruben Lozano-Aguilera, Cecile Nguyen, Smita Rao, Amber Tanaka, Brooke Vlahos, Peter Clark, Doug Downey, Yoav Goldberg, Ashish Sabharwal, Daniel S. Weld

arXiv:2510.21652v2cs.AIcs.CL

TL;DR

Existing agent benchmarks lack controlled tools, confounder accounting, standardized interfaces, holistic scientific-research coverage, and comprehensive baselines. AstaBench addresses these gaps with a broad benchmark, reproducible scientific-research environment, standardized agents, and evaluation of 57 agents across 22 classes, finding that science research assistance remains far from solved.

  • Problem

    Existing agent benchmarks often lack realistic controlled tools, confounder accounting, standardized interfaces, holistic scientific-research measures, and comprehensive baselines.

  • Method

    AstaBench provides over 2400 scientific-research problems, a standardized task interface, production-grade search tools, and standardized agent suites for controlled evaluation.

  • Results

    Despite meaningful progress on many fronts, evaluation of 57 agents across 22 classes finds that science research assistance remains far from solved.

  • Takeaways & Limitations

    AstaBench enables holistic measurement of current LLM-agent capabilities for scientific research assistance and supports continuous improvement.

  • Takeaways & Limitations

    The evaluation toolkit excludes latency-related discounts from its normalized cost calculations and relies on a frozen community-sourced pricing snapshot.

Abstract

from arXiv · show

AI agents hold the potential to revolutionize scientific productivity by automating literature reviews, replicating experiments, analyzing data, and even proposing new directions of inquiry; indeed, there are now many such agents, ranging from general-purpose "deep research" systems to specialized science-specific agents, such as AI Scientist and AIGS. Rigorous evaluation of these agents is critical for progress. Yet existing benchmarks fall short on several fronts: they often (1) lack reproducible agent tools necessary for a controlled comparison of core agentic capabilities; (2) do not account for confounding variables such as model cost and tool access; (3) do not provide standardized interfaces for quick agent prototyping and evaluation; (4) fail to provide holistic, product-informed measures of real-world use cases such as science research; and (5) lack comprehensive baseline agents necessary to identify true advances. In response, we define principles and tooling for more rigorously benchmarking agents. Using these, we present AstaBench, a suite that provides a holistic measure of agentic ability to perform scientific research, comprising 2400+ problems spanning the entire scientific discovery process and multiple scientific domains, and including many problems inspired by actual user requests to deployed Asta agents. Our suite comes with the first scientific research environment with production-grade search tools that enable controlled, reproducible evaluation, better accounting for confounders. Alongside, we provide a comprehensive suite of nine science-optimized classes of Asta agents and numerous baselines. Our extensive evaluation of 57 agents across 22 agent classes reveals several interesting findings, most importantly that despite meaningful progress on certain individual aspects, AI remains far from solving the challenge of science research assistance.

1 INTRODUCTION

AstaBench addresses shortcomings in existing agent benchmarks by providing a holistic, controlled, cost-aware evaluation suite for scientific research assistance. It combines broad scientific tasks, reproducible tools, standardized interfaces, and comprehensive agent baselines, while experiments show that science research assistance remains far from solved.

  • Motivation: Existing benchmarks often lack controlled tools, cost accounting, standardized interfaces, holistic scientific tasks, and comprehensive baselines.These gaps make comparisons vulnerable to differences in information access, tool usage, and computation.
  • AstaBench: AstaBench covers over 2400 problems across the scientific discovery process, multiple domains, and skills including literature understanding, planning, search, coding, and tool use.Many problems are based on real user requests, and the suite provides a standardized task interface for general agents.
  • Evaluation infrastructure: The Asta Environment provides production-grade search tools for controlled, reproducible retrieval from a large scientific-literature corpus.The environment is designed to separate core agentic abilities from differences in information access.
  • Evaluation infrastructure: The agent-eval toolkit and AstaBench Leaderboard account for tool usage and inference cost when reporting agent performance.The toolkit computes normalized costs from logged model usage, while the leaderboard incorporates these confounders into evaluation.
  • Agent baselines: The agent-baselines suite includes nine science-optimized Asta agent classes alongside numerous baselines for standardized comparison.Together with experiments spanning 57 agents across 22 agent classes, these resources support broad evaluation of scientific research agents.
  • Findings: Across the evaluated agents, meaningful progress appears on many fronts, but science research assistance remains far from solved.AstaBench is intended to provide a current snapshot and support systematic assessment as new agents are developed.

2 RELATED WORK

AstaBench builds on holistic agent evaluations and science-specific benchmarks, addressing gaps in task coverage, cost analysis, standardized tools, and agent-class breadth.

  • Holistic Agent Evaluations: Unlike related suites, AstaBench combines holistic science coverage with product-derived tasks, serious cost consideration, decoupled standard tools, and many agent classes.
  • Science Benchmarks and Agents for Science: Existing science benchmarks often target narrower sub-problems, such as coding, data analysis, research reproduction, or specific scientific domains.
  • AstaBench spans many scientific task categories and provides the most comprehensive evaluation of scientific agent performance to date.
  • Science Benchmarks and Agents for Science: AstaBench adds an end-to-end task that evaluates research projects from an initial idea through a written report and code.

3 ASTABENCH: A HOLISTIC SCIENTIFIC RESEARCH BENCHMARK SUITE

AstaBench is a holistic, reproducible benchmark suite for scientific research agents, with standardized scoring across diverse tasks and scientific capabilities.

  • AstaBench is presented as the first benchmark suite for holistic evaluation of agents’ ability to perform scientific research.
  • Its reproducibility is supported by realistic search tools, a standard benchmark interface, time-invariant cost reporting, and compatibility with new general agents.
  • The suite contains 11 benchmarks covering literature understanding, code and execution, data analysis, and end-to-end discovery.
  • AstaBench uses both LLM-based judging and programmatic evaluation, depending on the benchmark.

4 ASTA ENVIRONMENT

The Asta Environment provides reproducible research tools, evaluation infrastructure, cost accounting, confounder reporting, and a broad suite of compatible baseline agents.

  • The Asta Environment is described as the first realistic, reproducible scientific research environment for agents, combining standardized tools, evaluation infrastructure, a leaderboard, and numerous agents.
  • 4.1 STANDARD TOOLS FOR AGENTS: Its standard tools include date-restricted scientific-literature search and a stateful computational notebook for executing code incrementally.
  • 4.1 STANDARD TOOLS FOR AGENTS: Tools are decoupled from agents and integrate through MCP, while sandboxed code can call host-environment tools for CodeAct-style evaluation.
  • 4.2 AGENT-EVAL EVALUATION TOOLKIT & ASTABENCH LEADERBOARD: The agent-eval toolkit converts logged model usage into normalized dollar costs using a frozen pricing snapshot and cache discounts, but excludes latency-related discounts.
  • 4.2 AGENT-EVAL EVALUATION TOOLKIT & ASTABENCH LEADERBOARD: Leaderboards categorize submissions by agent openness and tooling control, and provide interactive plots and tables through a web interface.
  • 4.3 AGENT-BASELINES AGENTS SUITE: The agent-baselines Agents Suite supplies agents from 16 classes through a standard Inspect-compatible interface.

5 EXPERIMENTS

Across 57 agents and 22 classes, experiments show meaningful progress in literature understanding but low overall performance and persistent bottlenecks in coding, data analysis, and end-to-end discovery.

  • The best open-source agent with open-weight models scores 11.1% overall, while Asta v0 reaches 53.0%; even strong commercial agents cannot cover the full task range.
  • Replacing the underlying model is not uniformly beneficial: gpt-5 helps ReAct but lowers performance for several specialized agents, while its gains over o3 are usually 0%-5%.
  • Literature Understanding: Literature understanding is comparatively strong, with top literature question-answering systems scoring about 85% or higher on ScholarQA-CS2.
  • Literature Understanding: Literature table generation remains weak, with recall around 43%, although Asta Table Synthesis with gpt-5 wins and its gpt-5-mini version costs 13% as much.
  • Code and Execution: Coding and execution remain bottlenecks: all but two agents score below 25% on SUPER-Expert, with ReAct using gpt-5 at 41% and gpt-5-mini at 37%.
  • Data Analysis: Data analysis remains unsolved, with the maximum score reaching only 34%.
  • End-to-End Discovery: End-to-end completion remains near zero despite per-step scores up to about 70%, because roughly 10 sequential steps reduce the implied success rate to about 3%.

6 CONCLUSION AND FUTURE WORK

The paper presents AstaBench as a rigorous, holistic benchmark for scientific research and reports that agentic AI remains far from solving scientific research assistance despite progress on individual aspects.

  • 6 CONCLUSION AND FUTURE WORK: 57 agents across 22 architectural classes were evaluated on AstaBench, revealing meaningful progress on individual aspects but substantial distance from solving scientific research assistance.The evaluation used the agent-baselines Agents Suite and AstaBench benchmark suite.
  • 6 CONCLUSION AND FUTURE WORK: AstaBench combines a realistic, controlled scientific research environment with standardized agents and tools for more rigorous comparison.The Asta Environment provides controlled search tools, while the agent-eval toolkit powers the leaderboard.
  • 6 CONCLUSION AND FUTURE WORK: Future work targets performance-cost frontiers, open scientific-research agents, improved context management, and techniques for automated scientific discovery.The authors describe ongoing development of specialized techniques, tools, and open models.

ETHICS STATEMENT

The work reports legal review, consent for new real-world user queries, attribution for adapted benchmarks and leveraged agents, and collaboration with agent creators to support fair measurement.

  • ETHICS STATEMENT: Legal review covered all material presented in the work, and new real-world user queries were collected with user consent.
  • ETHICS STATEMENT: The authors credit adapted benchmarks and leveraged agents, and worked with creators where possible to measure existing agents fairly.Examples include Elicit, Future House, and SciSpace.

REPRODUCIBILITY STATEMENT

AstaBench emphasizes reproducibility through open-source artifacts, experiment logs, commit tracking, and date-restricted access to its supporting scientific literature corpus.

  • REPRODUCIBILITY STATEMENT: AstaBench releases open-source code for benchmarks, agents, and core infrastructure, together with logs of reported experiments.
  • REPRODUCIBILITY STATEMENT: The framework records specific repository commits, including data commits, to make reported experiments traceable.
  • REPRODUCIBILITY STATEMENT: Date-restricted access to the supporting document corpus is provided through the agent tools to improve reproducibility.
  • REPRODUCIBILITY STATEMENT: AI-based tools assisted with result-data analysis, plot and table code, error and reference checks, and minor writing assistance.

A PRINCIPLES FOR BENCHMARKING AGENTS

The proposed principles require agent benchmarks to reflect real-world complexity, use standardized reproducible environments, control confounders, support general-agent integration, and include comprehensive baselines.

  • A PRINCIPLES FOR BENCHMARKING AGENTS: Benchmark task suites should represent real-world usage by covering a broad range of relevant tasks informed by product usage where possible.The principle notes that product-usage data is often unavailable to academic benchmark creators.
  • A PRINCIPLES FOR BENCHMARKING AGENTS: Standard, realistic, and reproducible environments and tools should accompany benchmarks to enable controlled comparisons of agent capabilities.
  • A PRINCIPLES FOR BENCHMARKING AGENTS: Reporting should account for computational cost and tool usage so performance gains can be separated from repeated trials or privileged information access.
  • A PRINCIPLES FOR BENCHMARKING AGENTS: Standardized task interfaces should make it easier to integrate general agents across diverse tasks without task-specific adaptation bias.
  • A PRINCIPLES FOR BENCHMARKING AGENTS: Comprehensive agent baselines with standard interfaces are needed to identify state-of-the-art systems and provide strong starting points for future development.

B EVALUATION TOOLKIT: OPENNESS AND TOOLING

The toolkit classifies agents by openness and tooling to make evaluation transparency, reproducibility, and execution constraints explicit.

  • Openness: Agent openness ranges from open-source, open-weight systems to closed systems available only through a user interface.The categories distinguish full reproducibility, partial reproducibility, API-based verification, and no programmatic access.
  • Tooling: Agent tooling ranges from predefined evaluation-environment tools to custom interfaces and fully custom tools.For AstaBench, literature access and code execution are constrained by task-relevant portions of the Asta Environment.
  • Evaluation reporting: The results tables encode openness and tooling alongside macro-average performance and Pareto-optimal cost-performance status.Table 4 covers agents that can solve all tasks, while Table 5 reports literature-search results; bold entries mark Pareto-optimal column pairs.

C SUPPORTING EXPERIMENTAL RESULTS

The supporting results provide supplemental category and overall tables, using standardized labels for openness, tooling, uncertainty, and Pareto-optimality.

  • Results coverage: The supplemental results cover overall agents and category-specific performance for literature understanding, code and execution, data analysis, and end-to-end discovery.Additional tables report literature QA, literature-table generation, code, data-analysis, and end-to-end discovery results.
  • Overall results: Overall results include agents that attempt all benchmarks, while category results include agents able to solve all tasks within at least one category.The overall table reports macro averages over benchmark statistics.
  • Reporting conventions: Table annotations identify openness, tooling, 95% confidence intervals, and agents on the Pareto-optimal frontier.Openness applies to the agent and model used, while bold marks Pareto-optimal column pairs.

D FULL EXPERIMENTAL RESULTS

The full results report scores with costs and uncertainty across all tested configurations, including score-cost Pareto frontiers for scientific research benchmarks.

  • Full configuration results: Full results include all tested agent configurations and score-versus-cost plots showing the Pareto frontier, defined as the best agent for a given cost.The supplementary results extend the best-agent results from Section 5 and include 95% confidence intervals.
  • Statistical methodology: 95% confidence intervals are computed as ±1.96 × standard error, with category-level errors propagated through weighted averaging.The propagation assumes task independence, which could slightly underestimate uncertainty.
  • Score-cost analysis: The score-cost figures use mean scores with 95% confidence intervals and a logarithmic cost axis, while frontier points mark optimal quality-cost trade-offs.The figures apply these conventions across literature understanding, code and execution, data analysis, and end-to-end discovery.
  • Benchmark coverage: The results span literature search, literature QA, literature-table generation, code and execution, data analysis, and end-to-end discovery.Tables and figures cover PaperFindingBench, LitQA2, ScholarQA-CS2, ArxivDIGESTables-Clean, code benchmarks, DiscoveryBench, and E2E-Bench.

E EVALUATIONS

AstaBench evaluates scientific research assistance across search, literature understanding, coding, data analysis, and complete research cycles using diverse task designs and scoring methods.

  • Literature understanding: Literature-understanding tasks test retrieval, question answering, long-form synthesis, and structured comparison of related papers.ArxivDIGESTables-Clean evaluates whether generated tables entail reference-table statements after table unrolling.
  • Code and execution: Code and execution benchmarks test repository-based experiment setup, paper-linked reproduction, and routine data-science coding tasks.SUPER-Expert requires cloning, installing, configuring, running, and reporting results from low-resource research repositories; CORE-Bench-Hard− uses CodeOcean capsules.
  • Data analysis: DiscoveryBench evaluates data preparation, statistical analysis, transformation, and modeling for hypothesis discovery across datasets from six domains.Gold hypotheses are rigorously verified by data-analysis experts.
  • Benchmark suite: The evaluation suite spans challenging paper search, biomedical question answering, long-form scientific answers, literature-review tables, code execution, data analysis, and end-to-end research.E2E-Bench covers ideation, planning, experiment design, implementation, execution, analysis, and final reporting.
  • PaperFindingBench: PaperFindingBench expands paper search beyond semantic similarity to include navigational, metadata, and diverse semantic queries based partly on realistic information needs.Its query mix includes 48 navigational, 43 metadata, and 242 semantic queries, with some queries drawn from usage logs.

E.3 SCHOLARQA-CS2

ScholarQA-CS2 evaluates long-form scientific answers using multiple dimensions of answer quality, with LLM-based scoring and rubric-derived coverage. Its validation addresses agreement with experts and possible bias from sourcing rubric ingredients from evaluated systems.

  • Evaluation: ScholarQA-CS2 averages citation recall, citation precision, answer relevance, and answer coverage into one answer-quality score.Citation recall measures claim support, citation precision measures citation support, relevance measures paragraph-question alignment, and coverage measures necessary points included.
  • Evaluation: The evaluation uses an LLM judge to assess citation support, relevance, and coverage, with quotes enabling finer-grained citation scoring.Claims or citations without supporting quotes can receive intermediate scores when support is inferred from paper titles.
  • Coverage rubric: Answer coverage is computed from candidate ingredient clusters scored 0–2, with answer-critical ingredients weighted twice as heavily as valuable ingredients.Candidate ingredients are gathered from evaluated systems and converted into rubrics used during answer evaluation.
  • Data: The evaluation uses 100 computer-science questions from OpenSciLM as its test set and retains 100 ScholarQA-CS questions as a development set.The test questions were filtered for language, quality, and topic.
  • Validation: Holding systems out of rubric-ingredient extraction caused significant average drops of 2.5 points for three systems, but insignificant drops below 1 point for two others.The held-out systems were Asta Scholar QA, Elicit, SciSpace Deep Review, OpenAI Deep Research, and Perplexity Sonar Deep Research.

F.3.1 QUERY ANALYSIS

PaperFinder analyzes queries into structured search properties and routes them through specialized retrieval workflows. Its pipeline combines parallel retrieval, aggregation, relevance judgment, citation tracking, and follow-up reformulation.

  • Query analysis: The query analyzer separates navigational intent, semantic criteria, relevance criteria, and metadata constraints for downstream search components.It distinguishes requests for specific papers from requests for sets of papers and decomposes semantic queries into weighted subcriteria.
  • Limitations: The analyzer does not support author-affiliation metadata, and currently ignores adjectives such as “good paper” that lack metadata correlation.The paper notes that some ignored phrases may occasionally enter semantic criteria.
  • Navigational queries: PaperFinder handles specific-paper requests with three parallel strategies: title-API lookup, LLM-grounded title lookup, and citation-based candidate extraction.The resulting candidates are merged before subsequent processing.
  • Semantic queries: Semantic retrieval generates query rephrasings, aggregates snippets by paper across result sets, and adds cited papers as additional paper candidates.Each paper item can contain evidence from multiple queries and from papers that cite it.
  • Relevance filtering: Candidate papers undergo LLM relevance filtering, with the system retaining the most promising results and modeling query sources as a multi-armed bandits problem.The retrieval process broadens across successive steps while using relevance judgments to guide expansion.
  • Follow-up search: Follow-up queries are reformulated from relevant boundary papers, using their titles, abstracts, snippets, and the original query to broaden search.Boundary papers are relevant results farthest from the query in embedding space.

F.9 VALIDATION OF END-TO-END DISCOVERY AGENTS

The end-to-end discovery evaluation scores each task with a rubric across generated papers, code, and artifacts. This three-facet procedure improves robustness but still yields near-zero full-task completion because requirements compound.

  • Scoring: End-to-end scoring uses task-specific rubrics whose items are judged across the generated paper, code, and artifacts, then checked for cross-facet consistency.The overall score averages final rubric-item verdicts after consistency is assessed.
  • Scoring: The three-facet scorer reduces false positives and false negatives by comparing claims in the paper against implementation and artifact evidence.A paper may claim an experiment was run while code contradicts it, or omit a criterion that code demonstrates.
  • Validation: Human spot checks judged 92% of rubric items correct on a 50-item development sample, while identifying occasional over-crediting and missed conceptual nuances.These failures include vague paper mentions receiving credit and technically implemented code missing an important experimental detail.

G.2 EVALUATION ON FULL SET OF LITQA2 DATASET

The full LitQA2 evaluation tests PaperFinder on retrieval and question-answering with the benchmark’s original access conditions. PaperFinder performs similarly to the prior agent despite limited full-text availability and scores significantly higher on the reported comparison.

  • Setup: For PaperFinder, the evaluation supplies the question text without multiple-choice options and applies no task-specific modification or tuning.This preserves the agent’s general paper-finding interface for the task.
  • Dataset: The full LitQA2 dataset contains 199 queries, with 114 represented only by abstracts in the search index.The paper separately emphasizes LitQA2-FullText-Search as its main full-text evaluation subset.

H EVALUATION TASK SAMPLES AND PROMPTS

The evaluation samples specify research tasks, output formats, and scoring rubrics across literature search, report generation, and research-problem design. Together, they illustrate how AstaBench operationalizes scientific research assistance through structured prompts and criteria.

  • H.1.1 EXAMPLE PROBLEM: AstaBench examples span literature retrieval, report generation, coding-oriented research problems, and novel research-problem proposals.
  • H.1.1 EXAMPLE PROBLEM: Search tasks require comprehensive yet efficient retrieval, allow up to 250 papers, and distinguish known-paper requests from general queries.
  • H.1.1 EXAMPLE PROBLEM: Retrieval outputs use JSON fields for paper identifiers and concise markdown evidence supporting each paper’s relevance.
  • H.3.1 EXAMPLE PROBLEM: Report-generation tasks require structured JSON sections, inline citations, and reference snippets whose identifiers appear in the generated text.
  • H.3.2 EXAMPLE RUBRIC {: The rubric evaluates explanations of hallucination and bias through criteria covering definitions, shared training-data causes, direct links, and propagation.
  • H.3.2 EXAMPLE RUBRIC {: Rubric examples connect biased or incomplete data with hallucinations, including correlations of 0.81-0.83 between intrinsic bias and extrinsic hallucination rates.
  • H.10.1 EXAMPLE PROBLEM: A research-problem example describes enhancing a static DAG-ERC model with content-based edge selection for conversation emotion recognition.
  • H.11.1 EXAMPLE PROBLEM: Another proposed problem combines Complexity-Based Prompting with Imitation Demonstration Learning to improve language-model generalization on unseen tasks.
Loading 2510.21652v2…