Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

• Existing benchmarks often use non-scientific tasks or simplified question-answering and coding, while many scientific benchmarks are narrow in domain and lack clear difficulty definitions.The introduction argues these limitations leave current evaluations poorly suited to assessing agents’ capacities on core scientific tasks.Source: §1 · p. 2
• SciAgentArena covers five fields and evaluates challenging, real-world research tasks with multi-step workflows, verifiable intermediate states, tool use, environment interaction, and varying difficulty.Its domains are single-cell omics, spatial omics, computational drug discovery, EHR modeling, and genetics; tasks also test feasibility judgments.Source: §1 · p. 2
• SciAgentArena separates agent-specific running environments from evaluation, unifies input and output settings, and measures performance beyond precision, including stability, reliability, and cost.The framework also supports community submissions of agents, tasks, and evaluation metrics as a living benchmark platform.Source: §1 · pp. 2–3
• Agents analyze datasets effectively with fixed pipelines, but no agent dominates every task and their optimization and novel-discovery abilities remain limited.The study also reports heterogeneous contributions across data loading, analysis, optimization, and discovery.Source: §1 · p. 3
• Agents show inactive self-exploration, convergent method selection, shared errors, and—in some biomedical tasks—instability and sycophantic execution without validation.The authors suggest expanding knowledge bases and providing more detailed prompts to improve agents’ abilities on challenging scientific problems.Source: §1 · p. 3

2.1. Benchmark Overview.

• The benchmark covers five domains and 18 agents, organizing scientific challenges into Data Analysis, Optimization, Discovery, and Validity and evaluating agents at maximal functionality with suitable backbones.The five domains are Drug Discovery, Single-Cell Omics, Spatial Omics, EHR Modeling, and Genetics.Source: §2.1 · p. 3

2.2. Benchmarking AI Agents for Computational Drug Discovery.

• ToolUniverse performs strongest in molecular preprocessing, but agents falter on physiological formal-charge cases and target identification, as well as standardization, molecular interpretation, and required output schemas.Many compute properties on raw rather than standardized parent structures; all agents score highly on easy and medium charge cases, but harder physiological cases drop sharply.Source: §2.2 · pp. 6–7, Figure 3 · p. 11
• Claude Code and ToolUniverse lead multi-step local drug-data analysis; most leading agents handle standard curation, but external lookups and reconciliation of visual, identifier, and schema demands remain difficult.Multimodal agents approach perfect scores on figure-reading tasks, while weaker chemistry agents drop sharply; external database lookups are consistently weakest.Source: §2.2 · p. 7, Figure 3 · p. 11
• Drug-discovery method choices favor familiar workflows over task-specific strategies: agents usually handle rule-based PAINS filtering better than learned-model tasks such as hERG prediction.Molecule optimizers commonly use local search instead of the offered sample-efficient Bayesian optimization; incomplete catalogs and generic workflows also cause failures.Source: §2.2 · pp. 7–8
• Most agents solve single-objective molecular-design tasks, but none solves Valsartan SMARTS; multi-objective optimization exposes fragility under the 100-call budget and conjunctive constraints.Claude Sonnet 4.6 and Claude Code lead the category, and Claude Code’s sandboxed test-and-fix loop lets it solve Osimertinib MPO when the single-shot Sonnet run crashes.Source: §2.2 · pp. 8–9, Figure 3 · p. 11
• Validation remains unstable: agents may miss mixed IC50 units despite a greater-than-23,000 max/min ratio, and run-to-run variation is greatest on deeper chemistry and data-integrity judgments.CACTUS, Gemini 3 Pro, and Claude Code perform strongest overall across the validity suite, while ChemCrow is weakest; Gemini is most consistent among broadly evaluated agents.Source: §2.2 · pp. 9–10, Figure 3 · p. 11

2.3. Benchmarking AI Agents for Processing Cellular and Tissue-Level Signals.

• In single-cell preprocessing, Claude Code, ToolUniverse, and STELLA (mem) pass all tests, while spatial step-wise preprocessing is passed completely only by GPT-5.2; pipeline execution is harder in both settings.For spatial neighborhood graph construction, only GPT-5.2, STELLA (mem), and Biomni correctly account for graph symmetry; executable code can still fail scientific intent, as in Biomni’s label-reuse case.Source: §2.3 · pp. 11–12

Optimization (Model Selection): agents can choose established methods, but they converge

• For omics method selection, STELLA (mem) and Gemini 3 Pro lead spatial tasks, while Claude Code, STELLA (mem), and Codex lead single-cell tasks, often choosing familiar methods over specialized alternatives.Implementation failures calling or configuring methods are common, especially in pipeline mode; single-cell agents converge on Leiden, Harmony, and Wilcoxon, with some choices potentially mismatched to the data.Source: Optimization (Model Selection): agents… · pp. 12–13
• Across cell and tissue tasks, agents perform best on data analysis, converge on popular methods in selection, struggle with solution design, and remain unreliable at validation.Spatial agents commonly choose Harmony or scVI for batch correction, Leiden for spatial domains, and Global Moran’s I for spatially variable genes; familiar methods are favored over specialized alternatives. In OOD perturbation prediction, only Claude Code and STELLA (mem) produced runnable gene-interaction models that outperformed scLAMBDA (61) and pert-mean.Source: Optimization (Model Selection): agents… · pp. 13–14

2.4. Benchmarking AI Agents on Processing Patient-Level Information.

• In EHR tasks, STELLA (mem) led the 435-case stepwise workflow with F1 = 0.855, while Gemini 3 Pro led drug management with F1 = 0.311.For FHIR retrieval, STELLA (mem) scored F1 = 0.913 and Gemini 3 Pro 0.901; for rare disease diagnosis, STELLA (mem) led with F1 = 0.816. Drug management was the most difficult EHR task for every agent, and high diagnostic action overlap did not guarantee clinically executable reasoning.Source: §2.4 · pp. 16, 18, 19
• In statistical genetics, Claude Code and STELLA (mem) passed all 14 single-ancestry PRS subtasks and all 18 multi-ancestry subtasks, while MRAgent passed all 15 MR subtasks.Agents largely selected PRSCS (67) and PRS-CSx (68), but implementation failures clustered in infrastructure-heavy steps, ancestry harmonization, reproducible indexing, and final MR evidence synthesis.Source: §2.4 · pp. 16–18
• In statistical genetics, Claude Code alone scored 10/10, correctly completing all five valid tasks and rejecting all five false-premise tasks; omics agents differed in recognizing infeasible tasks.Claude Code identified the invalid CNV premise, whereas STELLA proceeded; for spatial-neighborhood enrichment, STELLA checked package capability while Biomni assumed the task was executable.Source: Optimization (Model Selection): agents… · p. 14, §2.4 · p. 19
• F1 = 0.844 for STELLA (mem), the highest EHR average, while Claude Sonnet 4.6 scored 0.449, Gemini 3 Pro 0.444, and TxAgent-8B 0.038.STELLA’s advantage was driven largely by the stepwise workflow task; all agents scored poorly on drug management, exposing a shared difficulty with pharmacological reasoning.Source: §2.4 · p. 20

2.5. Benchmarking AI Agents on Cross-Domain Tasks.

• The cross-domain evaluation comprises four tasks, including eQTL computation, drug-target identification, and synthetic-lethality prediction, spanning genetics, omics, drug discovery, and EHR modeling.The eQTL task uses GTEx v8 data to test whether agents can generate code to read RNA-seq and WGS profiles and compute eQTL.Source: §2.5 · p. 20
• Claude Code identified more database-overlapping eQTLs, GPT-5.2 achieved the highest PCC_beta, and all reported agents had high variability and often omitted covariates.Nearly all agents chose OLS; agents not reported could not generate successfully running code, indicating difficulty processing multimodal data.Source: §2.5 · p. 22
• Medea led synthetic-lethality prediction and ToolUniverse led target-gene identification, while general-purpose LLMs did not outperform domain-expert agents on cross-domain tasks.The evaluation sampled 20 tasks; none of the agents achieved particularly high scores, and the authors identify dataset reading and parsing as key capabilities.Source: §2.5 · p. 22

2.6. Error Analyses and General Suggestions Summarized from Different Scientific Fields.

• Across domains, failures often arose from unverified representations, assumptions, references, or generic methods rather than coding alone.In drug discovery, examples included molecular standardization and charge-state errors, brittle constrained searches, incomplete safety coverage, and over-interpreting invalid premises; reliable agents must repeatedly check data, tools, representations, and conclusions.Source: §2.6 · pp. 22–23
• Non-existent code accounted for 48.6% of spatial-omics errors, while running errors and wrong-function selection each accounted for 33.9% of single-cell errors.Spatial failures also included runtime and graph-structure errors; single-cell failures included package incompatibilities, unsuitable data structures or functions, task misalignment, and misunderstood constraints.Source: §2.6 · p. 23
• EHR agents struggled with over-generation, under-production, and incomplete medication sets, while genetics failures centered on multi-stage workflow assembly, API grounding, and advanced analyses.Genetics validation also exposed false acceptance of unsupported assumptions, weak causal reasoning, compositional-prompt failures, and answers insufficiently grounded in the supplied data.Source: §2.6 · pp. 23–24
• The authors recommend stronger tool grounding, API verification, persistent state tracking, context inspection, assumption checks, and refusal of invalid tasks.They advise greatest confidence for well-specified data-analysis workflows, more caution for model selection and optimization, and mandatory human oversight for validation-sensitive tasks. Generalist agents including Claude Code, ToolUniverse, and STELLA appeared in the top three multiple times, while domain-specific agents also excelled on some home-territory subtasks.Source: §2.6 · pp. 24–25

Discussion

• SciAgentArena combines challenging, step-wise verifiable scientific tasks with broad evaluation across major domains to address gaps in existing agent and science benchmarks.It also includes tasks for assessing agent limits and cross-domain integration.Source: §3 · p. 26
• Across five domains, agents perform best on well-specified data-analysis workflows, but struggle with longer-horizon planning, discovery, optimization, and validity checking.They handle standard procedures in areas such as cheminformatics, omics, EHR retrieval, and genetics, while encountering difficulties including rare-disease diagnosis, spatial-specific reasoning, multi-ancestry integration, and mechanistic safety reasoning.Source: §3 · p. 27
• Exploration, conservative method choice, poor state tracking, tool hallucinations, weak domain validation, and compliance with scientifically invalid requests are common agent failure modes.The discussion identifies these issues as targets for error analysis and improvement.Source: §3 · p. 27
• Future AI scientists should assess tasks more rigorously, understand and match tools to them, choose within available resources, and support human interaction in special circumstances.These capabilities are presented as a roadmap for addressing the identified issues.Source: §3 · p. 27
• SciAgentArena’s modular design separates code execution from evaluation, allowing agent extensions and new tasks to be integrated simultaneously.The authors aim for it to become a general testing ground for agent design in scientific research.Source: §3 · p. 27
• The benchmark’s expert-selected tasks may introduce bias, and its scope should expand to domains such as physics and materials science and emerging agent paradigms such as multimodal systems.The discussion calls for regular updates as agent frameworks evolve.Source: §3 · p. 27

4.1. Framework

• The framework combines an extensible dataset with coordinated agent runtime and a shared evaluation environment, emphasizing verifiable multi-step workflows and expert-defined, task-specific scoring.It supports varied agent outputs, produces task metrics and leaderboards, and offers a front end for task selection, uploads, and community contributions.Source: §4.1 · pp. 27–28

4.2. Domain: Computational Drug Discovery

• The drug-discovery benchmark defines 78 tasks across five categories: preprocessing, data analysis, molecule optimization, safety assessment, and claim validation.Evaluation ranges from reference-answer checks and analysis checkpoints to optimization success criteria and safety or claim-validity scoring; datasets are not used to train or fine-tune agents.Source: §4.2 · pp. 29–30

4.3. Domain: Single-Cell Omics

• The single-cell benchmark includes 12 analysis and algorithm-design tasks across preprocessing, method selection, and optimization, plus 15 questions testing whether analyses are executable.Analysis uses pass@1 checkpoints, while optimization uses task-specific metrics; datasets span human pancreas, PBMC, and heart, developmental trajectories, and perturb-seq data.Source: §4.3 · pp. 30–32

4.4. Domain: Spatial Omics

• The spatial-omics benchmark defines 11 preprocessing and method-selection tasks, plus 15 questions testing whether requested analyses are executable on the data.Scoring uses pass@1 checkpoints and task-specific metrics, including spatial-neighborhood agreement and silver-standard comparisons for spatially variable genes; three annotated tissue datasets are used.Source: §4.4 · pp. 32–33

4.5. Domain: Electronic Health Records

• The EHR benchmark covers four task types across 505 evaluable patient cases: FHIR workflows, stepwise clinical workflows, rare-disease diagnosis, and drug recommendation.It also specifies a target-trial-style causal analysis task requiring cohort construction, confounder adjustment, and reproducible artifacts, scored across eight dimensions with temporal validity as a hard gate.Source: §4.5 · pp. 33–34

4.6. Domain: Genetics

• The genetics benchmark covers single-ancestry and multi-ancestry polygenic risk score prediction, Mendelian randomization, and conceptual validation.Its motivating scope includes reproducible workflows and interpretation under assumptions about ancestry, linkage disequilibrium, confounding, and causal validity.Source: §4.6 · p. 35
• Single-ancestry PRS tasks (T1, n=14) require agents to build an end-to-end European-ancestry scoring pipeline and address common harmonization and variant-processing failure modes.The four stages cover method selection and input validation, variant QC and allele harmonization, weight estimation, and scoring with score QC.Source: §4.6 · p. 35
• The 187-pair MR benchmark (T2, n=15) tests causal-effect estimation across population stratification, horizontal pleiotropy, and established trait pairs using 13 MR methods.Agents harmonize exposure and outcome summary statistics, select instruments through LD clumping, and return estimates, standard errors, 95% confidence intervals, and significance measures. Three methods are excluded for computational or resource constraints.Source: §4.6 · pp. 35–36
• Multi-ancestry PRS tasks (T3, n=18) test joint PRS methods across 22 traits and up to five UK Biobank ancestry cohorts with heterogeneous ancestry coverage.Agents must map ancestry-specific inputs, harmonize data, estimate joint weights, and score ancestry-matched cohorts while avoiding ancestry drops, N substitutions, and misapplied weights.Source: §4.6 · p. 36
• Genetics validation tasks assess whether agents answer five valid expert-designed questions correctly and reject five intentionally invalid or fabricated questions with plausible justification.The valid questions cover analytical assumptions, diagnostic checks, and interpretation challenges; the invalid ones test recognition of unsupported premises and biased assumptions.Source: §4.6 · pp. 36–37
• PRS, MR, and multi-ancestry PRS outputs are scored pass/fail at the subtask level, while validation responses are checked against expert-curated labels or for appropriate rejection.Criteria include execution, data processing, output formatting, reference results or accepted checks, and valid predictive-performance reporting; MR requires successful implementation and interpretation of at least one method, and multi-ancestry PRS requires ancestry-stratified outputs. PRS data use UK Biobank and 1000G LD references; MR summary statistics come from the specified GWAS datasets and consortia.Source: §4.6 · pp. 36–37

4.7. Cross Domain Tasks.

• Cross-domain tasks evaluate eQTL computation on GTEx v8, drug-target selection among five candidate genes, and synthetic-lethality prediction.eQTL evaluation uses 100 randomly selected genes, ground truth from GTEx, and F1, Jaccard, and beta PCC metrics; drug-target and synthetic-lethality tasks each subsample 20 Medea benchmark questions and use accuracy.Source: §4.7 · p. 37

5. Code and Data Availability

• The authors used OpenAI, Claude, and Gemini APIs and three university computing resources, and provide their code in the SciAgentArena GitHub repository.The computing resources were the Misha cluster at Yale, Yale Research Computing Center’s McCleary cluster, and Northeastern University’s Discovery Center.Source: §5 · p. 38
• All study datasets and the benchmark’s tasks, solutions, metadata, and configurations are publicly available on Hugging Face.Source: §5 · p. 38

• Tianyu Liu led coordination and planning with Yuanqi Du, Wengong Jin, Pan Lu, Zhuoran Yang, and Kaize Ding.Source: §6 · p. 38
• Tianyu Liu led framework design and development with Allen Xin Wang, Antonia Panescu, Lisa Xinyi Chen, Xinyu Wei, and Ziyao Zeng.Source: §6 · p. 38
• Allen Xin Wang and Antonia Panescu led drug discovery with Jihang Chen, Sihan Jiang, Ziqing Wang, and Siyi Gu.Source: §6 · p. 38
• Tianyu Liu and Wenxin Long led single-cell omics; Lisa Chen led spatial omics with Tianyu Liu and Wenxin Long.Source: §6 · p. 38
• Xinyu Wei led EHR with Siyu Chen and Xinyang Hu; Yueqian Jing led genetics with Haoran Shao, Leqi Xu, Wangjie Zheng, and Zhiyuan Cao.Source: §6 · p. 38
• Tianyu Liu led cross-domain work with Allen Xin Wang, Xinyu Wei, and Ada Fang; everyone drafted and edited the original draft, and Tianyu Liu and Hongyu Zhao supervised.Source: §6 · p. 38

8. Conflict of Interests

• The conflict-of-interest statement requires confirmation with all authors.Source: §8 · p. 38

A. Categories of AI Agents

• Agents are categorized by tool use, human-in-the-loop interaction, autonomous research, self-evolution, and domain specialization.The authors selected representative methods based on agent capabilities to balance rigor, timeliness, and experimental costs; Supplementary Table 2 compares agents across these dimensions.Source: A. Categories of AI Agents · p. 51, Table 2 · p. 51
• Most selected agents support multiple tool types, whereas self-evolving capability is less common.The selected agents cover flagship agent-design ideas and leading methods in their respective source domains.Source: A. Categories of AI Agents · p. 51
• Most agents cost no more than $20 per run, with Biomni as the exception; the evaluation also included memory-enhanced and basic STELLA modes.Agent costs, agent versions, and base-model versions are reported in Supplementary File X.Source: A. Categories of AI Agents · p. 51

B. Differences between our framework and other benchmark studies

• Domain-specific benchmarks focus on areas such as computational biology or medicine, while broader benchmarks often simplify settings and miss realistic tool-based, multi-step scientific workflows.Examples of domain-specific benchmarks include BixBench, BioAgentBench, BAISBench, CompBioBench, and MedAgentBench; broader examples include AstaBench, TerminalBench, HLE, and SDE.Source: B. Differences between our framework… · p. 52
• Most prior benchmarks offer only partial stepwise verification and extensibility, with limited difficulty control, diverse inputs, validity and robustness evaluation, or actionable suggestions.Many also restrict evaluation to narrow agent classes and omit confounders such as cost and openness.Source: B. Differences between our framework… · p. 52
• The proposed framework spans biology, chemistry, drug discovery, medicine, and cross-domain tasks while supporting stepwise verification, extensibility, difficulty labels, diverse inputs, and suggestion generation.It evaluates heterogeneous, tool-augmented, and multi-agent systems rather than only LLMs.Source: B. Differences between our framework… · p. 52
• Validity checks and richer evaluation signals assess intermediate reasoning and final outcomes, while the framework evaluates executability, costs, tools, openness, and robustness as performance confounders.Its larger and more diverse set of agent types is intended to provide a scalable, generalizable testbed across scientific discovery tasks.Source: B. Differences between our framework… · p. 52

C. Prompt list

• Agents received a file path to read before being given the task definition and requirements for formulating their inputs; outcome examples appear in each main figure.Task-specific prompts are included in the released dataset.Source: C. Prompt list · p. 52

D. Supplementary Figures

• Supplementary Fig. 1 presents agent performance on trajectory integration and the proportions of batch-effect-correction methods agents selected.Panel (a) compares performances across agents; panel (b) shows selected-method proportions.Source: Figure 1 · p. 53
• Supplementary Fig. 2 reports perturbation-prediction performance using PCC evaluated on the top 20 DEGs.Source: Figure 2 · p. 54
• Supplementary Figs. 3 and 4 report workflow-generation stability across three replicates, including step-wise and pipeline modes and a Spatial Omics pipeline analysis.Supplementary Fig. 3 covers step-wise and pipeline modes; Supplementary Fig. 4 shows pipeline-mode results based on Spatial Omics.Source: Figure 3 · p. 55, Figure 4 · p. 56
• Supplementary Fig. 5 examines agents’ identification of valid omics tasks through agent comparisons and correct-versus-incorrect case studies.It includes 15 single-cell-omics tasks, a single-cell case study, and a spatial case study.Source: Figure 5 · p. 57
• Supplementary Fig. 6 presents cross-domain results for eQTL computation, selected methods and covariates, SL prediction, and target identification.Source: Figure 6 · p. 58
• In T5 causal EHR analysis on MIMIC-IV Demo, Supplementary Fig. 7 reports perfect D3 temporal-validity scores for all agents and identifies D6 artifact reproducibility as most discriminating.Cells show weighted per-agent, per-dimension scores relative to a maximum of 200 total.Source: Figure 7 · p. 59
• Supplementary Fig. 8 shows error proportions for tasks based on single-cell and spatial omics data.Source: Figure 8 · p. 60