Source-linked AI summary
Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Tianyu Liu, Allen Xin Wang, Antonia Panescu, Lisa Xinyi Chen, Wenxin Long, Xinyu Wei, Yueqian Jing, Ziyao Zeng, Jihang Chen, Sihan Jiang, Ziqing Wang, Siyi Gu, Siyu Chen, Xinyang Hu, Haoran Shao, Leqi Xu, Wangjie Zheng, Zhiyuan Cao, Ada Fang, Botao Yu, Kunyang Sun, Rex Ying, Arman Cohan, Qingyu Chen, Lingzhou Xue, Kaize Ding, Yuanqi Du, Wengong Jin, Zhuoran Yang, Marinka Zitnik, James Zou, Hua Xu, Hongyu Zhao
TL;DR
Scientific AI-agent benchmarks rarely reflect the complexity and interactive demands of real research. SciAgentArena addresses this gap with approximately 200 verifiable tasks across scientific domains and finds that agents are useful for well-specified workflows but uneven on open-ended research.
Problem
Existing AI-agent benchmarks provide limited evidence about performance on complex, interactive, and scientifically grounded research tasks.
Method
SciAgentArena evaluates AI agents on approximately 200 scientifically grounded tasks with stepwise verification in an interactive, agent-agnostic environment.
Results
Agents perform best on well-specified data-analysis workflows but struggle with longer-horizon planning, optimization, discovery, and scientific validity checking.
Takeaways & Limitations
SciAgentArena provides a framework for measuring progress and identifying priorities for improving agents’ reliability, autonomy, and scientific reasoning.
Takeaways & Limitations
Agents remain fragile when optimization requires coordinating multiple constraints, especially for conjunctive multi-objective targets.
Abstract
from arXiv · showhide
AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.
1. Introduction
AI agents show promise for scientific research, but existing evaluations do not adequately capture realistic, heterogeneous, end-to-end scientific work. SciAgentArena addresses this gap with a practical, interactive benchmark designed to assess agents’ multi-step, verifiable, tool-using performance and diverse capabilities.
- Motivation: AI agents have contributed to complex real-world problems and show potential for literature reviews, experimental planning, and novel matter design.These capabilities motivate systematic evaluation in scientific research settings.
- Research questions: Scientific applications raise questions about agents’ capability, reliability across heterogeneous settings, and the benchmark design needed for fair comparison.The introduction highlights chemistry, healthcare, and life sciences as demanding application contexts.
- Benchmark gap: Existing benchmarks lag behind end-to-end scientific research because they often evaluate fixed models, tools, datasets, or specific tasks rather than realistic agent workflows.The passage identifies this as a central gap in current AI-agent benchmark studies.
- Benchmark design: A new benchmark should emphasize challenging, practical, scientifically grounded problems with multi-step workflows, verifiable intermediate states, tool use, and environment interaction.It should also span domains and difficulty levels while accounting for efficiency, cost, competence, adaptability, reliability, and responsibility.
- Benchmark platform: The proposed evaluation system extends assessment beyond precision to stability, reliability, and cost while supporting community submissions of agents, tasks, and evaluation metrics.The authors describe this as a living benchmark platform intended to improve evaluation efficiency and future agent design.
- Initial findings: No single AI agent dominates every task, revealing uneven collaboration and the need to improve agents’ generalization ability in scientific research.This result is presented as evidence of the benchmark’s difficulty and of distinct agent tendencies across science-related problems.
2. Results
Agents perform reliably on well-specified chemical preprocessing and standard dataset analyses, but struggle with benchmark-aligned molecular handling, physiological chemistry, adaptive method selection, and multi-constraint molecular optimization. Tool grounding and iterative code execution improve robustness when they match the task’s demands, yet familiar workflows and procedural brittleness remain widespread limitations.
- Data Preprocessing and Analysis: Agents reliably execute well-specified cheminformatics workflows, but fail on benchmark-aligned molecular representations, output formats, and physiological formal-charge reasoning.They often compute descriptors on raw inputs instead of intended parent structures and mishandle salts, fragments, isotopes, and heavy-atom-only structures.
- Data Preprocessing and Analysis: Tool grounding is particularly effective for preprocessing because well-defined operations map naturally onto chemistry tool calls, with ToolUniverse achieving the strongest category performance.Its chemistry tools can standardize molecules to parent structures and resolve normalization traps that defeat direct computation on raw inputs.
- Data Preprocessing and Analysis: Standard single-table analyses are strongest, while vision tasks divide performance between multimodal and other agents and require pixel-level interpretation of rendered scientific figures.Leading agents commonly solve SAR plotting, matched-molecular-pair analysis, bootstrap confidence intervals, temporal splitting, and lead prioritization.
- Model Selection: Agents choose plausible, familiar method families but rarely adapt them to task-specific objectives, constraints, benchmark-standard catalogs, or validation requirements.They favor genetic algorithms, beam search, hill-climbers, generic workflows, incomplete SMARTS libraries, and standard fragment-edit searches over more objective-aware strategies.
- Model Optimization: 100 oracle calls constrain molecule-optimization tasks, and performance drops sharply from single-objective to multi-objective settings for every evaluated agent.Single-objective tasks are usually solved, whereas multi-objective tasks expose weaknesses in coordinating several criteria simultaneously.
- Model Optimization: Iterative sandboxed execution improves optimization robustness, but agents remain vulnerable to crashes, open-loop search, and cheminformatics edge cases during candidate generation.Claude Code’s test-and-fix loop solves the Osimertinib MPO task after a single-shot Sonnet run crashes, while chemistry-specialized tools may remain unused when tasks require direct code execution.
3. Discussion
SciAgentArena shows that current AI agents can assist scientific research unevenly, performing best on well-specified workflows while struggling with longer-horizon, adaptive, and discovery-oriented tasks. Its modular benchmark design supports extensibility, but domain-selection biases and limited coverage remain important limitations.
- Findings: Current AI agents are useful but uneven scientific collaborators, performing best on well-specified Data Analysis workflows with known procedures.Examples include cheminformatics preprocessing, single-cell/spatial omics pipelines, FHIR retrieval, and standard genetics analyses.
- Findings: Agent reliability drops when tasks require longer-horizon planning, adaptive method selection, optimization, discovery, or validity checking.
- Future directions: Future agents should assess tasks more rigorously, understand and match required tools or skills, perceive available resources, and make decisions within resource constraints.
- Benchmark design: SciAgentArena’s modular design separates code execution from evaluation, enabling simultaneous integration of agent extensions and new tasks.The benchmark is intended as a general testing ground for agent design and may eventually become the “Leetcode” of scientific research.
- Limitations: Domain-expert task selection may introduce difficult-to-avoid biases, motivating broader coverage of fields such as physics and materials science.The authors also note that the evaluated agent categories are based on existing frameworks.
4. Methods
SciAgentArena combines a multi-domain task dataset with runtime and evaluation frameworks that support fair, interactive, agent-agnostic assessment. Its methods define domain-specific task sets, datasets, and evaluation metrics, including pass@1, scIB-based scores, and action-level F1 for clinical workflows.
- Benchmark framework: SciAgentArena comprises a dataset spanning three question types and five domains, plus runtime and evaluation frameworks operating in a shared environment for fair comparison.A front-end website lets users interact with the framework and upload agent outcomes for evaluation.
- Benchmark framework: The runtime framework safely runs frontier LLMs and representative agents, evaluating GPT 5.2, Claude Sonnet 4.6, and Gemini 3 Pro in every domain.Agents also use frontier LLMs as default backbones, although some require alternative settings because of compatibility or cost.
- Scientific task design: The dataset defines 78 tasks across five categories, including chemical data preprocessing workflows involving molecular property calculation, filtering, similarity ranking, conversion, target identification, and charge measurement.These tasks are designed to test reliable end-to-end cheminformatics under messy or ambiguous molecular inputs.
- Evaluation framework: Analysis-related tasks use pass@1 based on whether processed data passes all checkpoints, while batch-effect correction uses scIB-derived S_total metrics for batch reduction and biological information conservation.If no usable algorithm is produced, PCs are computed to evaluate batch-effect correction.
- Scientific task design: Pipeline-design datasets comprise human-annotated Pancreas, PBMC, and Heart scRNA-seq datasets, while trajectory integration uses datasets from HSC and B-cell developmental series.Heart is an atlas-level, large-scale dataset, and marker genes are collected from CellMarker 2.0.
- Clinical task design and evaluation: Clinical evaluation includes FHIR workflows (T1, n=20), stepwise workflows (T2, n=455), rare-disease diagnosis (T3, n=20), and drug recommendation (T4, n=10), using action-level F1 for T1–T3.T1–T3 scores are macro-averaged across cases using bidirectional substring matching; T4 produces an open-ended medication list scored for lexical fidelity.
5. Code and Data Availability
The study’s code is publicly available through its GitHub repository, while the complete SciAgentArena benchmark materials and datasets are hosted on Hugging Face.
- Implementation: The authors used OpenAI, Claude, and Gemini APIs, along with computing clusters at Yale and Northeastern University, for method development and experiments.The clusters included Misha at Yale’s Wu Tsai Institute, McCleary at Yale Research Computing Center, and Discovery Center at Northeastern University.
- Implementation: The study’s code is available in the SciAgentArena GitHub repository.The repository is linked at https://github.com/HelloWorldLTY/SciAgentArena.
- Data availability: All datasets used in the study are publicly available through the SciAgentArena benchmark collection on Hugging Face.The hosted materials include tasks, solutions, datasets, metadata, and configurations.
6. Author Contributions
The authors’ contributions span coordination and planning, framework design and development, and scientific application areas including drug discovery, single-cell omics, spatial omics, and EHR.
- Author Contributions: Contributions covered coordination and planning, framework design and development, drug discovery, single-cell omics, spatial omics, and EHR.Tianyu Liu led coordination and planning and framework design and development; listed domain leads included Allen Xin Wang and Antonia Panescu for drug discovery, Tianyu Liu and Wenxin Long for single-cell omics, Lisa Chen for spatial omics, and Xinyu Wei for EHR.
8. Conflict of Interests
The conflict-of-interests statement requires confirmation from all authors.
- Confirmation is needed from all authors regarding conflicts of interest.
A. Categories of AI Agents
The benchmark categorizes selected AI agents across five capability dimensions, spanning tool use, human interaction, autonomous research, self-evolution, and domain specialization. Agent versions, costs, and alternative operating modes are also documented and evaluated.
- Capability dimensions: AI agents are categorized by tool use, human-in-the-loop interaction, auto research, self-evolution, and domain specialization.These five aspects define the capability-based categories used to select representative methods for evaluation.
- Selection criteria: Representative evaluation methods were selected according to agent capabilities to maintain rigorous, timely conclusions while controlling experimental costs.
- Implementation details: Most agents cost no more than $20 per run, except Biomni, and the study reports agent and base-model versions.The evaluation also compares STELLA’s memory-enhanced and basic modes.
B. Differences between our framework and other benchmark studies
Existing AI-agent benchmarks are fragmented in scientific scope and evaluation rigor: domain-specific benchmarks lack breadth, while general benchmarks often simplify scientific workflows. SciAgentArena addresses these limitations through a unified, multi-domain framework with stepwise and realistic agent evaluation.
- Benchmark landscape: Existing benchmarks show clear fragmentation across scientific scope and evaluation rigor.The comparison considers eight dimensions, including scientific domains, stepwise evaluation, extensibility, and difficulty annotation.
- Benchmark landscape: Domain-specific benchmarks cover areas such as computational biology and medicine but typically lack breadth across disciplines.Examples include BixBench, BioAgentBench, BAISBench, CompBioBench, and MedAgentBench.
- Benchmark landscape: General benchmarks seek broader coverage but often prioritize simplified or LLM-only settings, limiting realistic workflow evaluation.These limitations affect workflows involving tools, heterogeneous data, and multiple scientific contexts.
- SciAgentArena framework: SciAgentArena spans biology, chemistry, drug discovery, medicine, and cross-domain tasks within one comprehensive evaluation framework.It supports stepwise verification, extensibility, difficulty annotation, diverse input formats, and suggestion generation.
- SciAgentArena framework: The framework emphasizes realistic agent evaluation beyond LLM-only settings while supporting heterogeneous scientific workflows.Its design addresses limitations in existing benchmarks by incorporating broader domains and richer evaluation capabilities.
C. Prompt list
Task-specific prompts are included in the released dataset and guide agents from file access through task execution. The paper also provides examples of agent outcomes in each main figure.
- C. Prompt list: Task-specific prompts provide agents with file paths, access instructions, task definitions, and input-formulation requirements.Examples of agent outcomes appear in each main figure.
D. Supplementary Figures
The supplementary figures extend evaluation across trajectory integration, perturbation prediction, workflow stability, task validity, cross-domain analysis, causal EHR analysis, and omics error distributions. They also include case studies illustrating correct and incorrect task-validity judgments.
- Trajectory integration performance and batch-effect-correction method selection are compared across AI agents.
- Perturbation prediction performance is evaluated using Pearson correlation coefficients based on the top 20 differentially expressed genes.The figure reports perturbation prediction performances (PCC) evaluated on the top 20 DEGs.
- Workflow-generation stability is analyzed across three replicates in step-wise and pipeline modes, including a pipeline-mode analysis for Spatial Omics.
- Agent abilities to identify valid omics-data tasks are assessed across single-cell and spatial omics, with correct and incorrect case studies presented.
- Cross-domain supplementary analyses cover eQTL computation, method and covariate selection, SL prediction, target identification, causal EHR scoring, and omics error distributions.The causal EHR analysis reports perfect temporal-validity scores for all agents and identifies artifact reproducibility as the most discriminating dimension.