Source-linked AI summary
ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, Huan Sun
TL;DR
Language agents are increasingly proposed for end-to-end scientific discovery, but evidence about their ability to perform individual real-world workflow tasks remains limited. ScienceAgentBench addresses this gap with a validated benchmark of executable scientific programming tasks and rigorous evaluation. Across three attempts, the best analyzed agent solved only 34.3% of tasks with expert knowledge, indicating that current agents do not yet automate data-driven discovery pipelines.
Problem
Evidence remains limited on whether language agents can reliably complete individual tasks in real-world scientific workflows before end-to-end automation claims are made.
Method
ScienceAgentBench evaluates agents on 102 tasks from 44 peer-reviewed publications across four disciplines using executable Python programs and expert-validated multi-axis metrics.
Results
34.3% SR was achieved by Claude-3.5-Sonnet with self-debug and expert-provided knowledge across three attempts, the best performance among the analyzed agents.
Takeaways & Limitations
Current language agents cannot yet automate essential data-driven discovery tasks or complete whole research pipelines end-to-end.
Takeaways & Limitations
Rubric-based human evaluation can overlook scientifically equivalent implementations and may vary subjectively on figure formatting, so successful programs may not receive perfect ratings.
Abstract
from arXiv · showhide
The advancements of large language models (LLMs) have piqued growing interest in developing LLM-based language agents to automate scientific discovery end-to-end, which has sparked both excitement and skepticism about their true capabilities. In this work, we call for rigorous assessment of agents on individual tasks in a scientific workflow before making bold claims on end-to-end automation. To this end, we present ScienceAgentBench, a new benchmark for evaluating language agents for data-driven scientific discovery. To ensure the scientific authenticity and real-world relevance of our benchmark, we extract 102 tasks from 44 peer-reviewed publications in four disciplines and engage nine subject matter experts to validate them. We unify the target output for every task to a self-contained Python program file and employ an array of evaluation metrics to examine the generated programs, execution results, and costs. Each task goes through multiple rounds of manual validation by annotators and subject matter experts to ensure its annotation quality and scientific plausibility. We also propose two effective strategies to mitigate data contamination concerns. Using ScienceAgentBench, we evaluate five open-weight and proprietary LLMs, each with three frameworks: direct prompting, OpenHands CodeAct, and self-debug. Given three attempts for each task, the best-performing agent can only solve 32.4% of the tasks independently and 34.3% with expert-provided knowledge. In addition, we evaluate OpenAI o1-preview with direct prompting and self-debug, which can boost the performance to 42.2%, demonstrating the effectiveness of increasing inference-time compute but with more than 10 times the cost of other LLMs. Still, our results underscore the limitations of current language agents in generating code for data-driven discovery, let alone end-to-end automation for scientific research.
1 INTRODUCTION
ScienceAgentBench addresses skepticism about end-to-end scientific automation by evaluating agents on individual, real-world workflow tasks. It introduces a validated benchmark designed to measure both capabilities and limitations before broader automation claims.
- Each benchmark task requires completing one or more sub-tasks successfully to achieve its overall goal.
- High-quality benchmarks for individual tasks in real-world scientific workflows are lacking, despite claims and interest in end-to-end automation.
- ScienceAgentBench contains 102 tasks drawn from 44 peer-reviewed publications across four scientific disciplines, validated by nine subject matter experts.
- The benchmark unifies every target output as a self-contained Python program and evaluates generated programs, execution results, and costs.
- Claude-3.5-Sonnet with self-debug solved 10.8% more tasks than OpenHands CodeAct while costing 17 times less API fees without expert-provided knowledge.
- Current language agents cannot yet automate essential data-driven discovery tasks or complete research pipelines end-to-end.
2 SCIENCEAGENTBENCH
ScienceAgentBench formulates scientifically grounded discovery tasks as executable Python code-generation problems and validates them through expert review and reproducibility checks. Its evaluation combines execution, task outcomes, figure quality, and fine-grained rubric scores.
- Benchmark formulation: ScienceAgentBench targets science co-pilots that generate executable Python programs for processing, analyzing, and visualizing data.
- Benchmark formulation: Each task includes an instruction, dataset information, optional expert-provided knowledge, and an annotated self-contained program.
- Task construction: Tasks are adapted from peer-reviewed code examples, with datasets collected and preprocessing and annotation performed as part of task construction.
- Validation: Nine subject matter experts validate task realism, scientific plausibility, and accompanying knowledge, while annotators reproduce programs and refine annotations.
- Evaluation: Evaluation measures valid execution, task success, figure quality, and program costs, with success criteria implemented as task-specific evaluation programs.
- Evaluation: Rubric-based evaluation scores Data Loading, Data Processing, Modeling or Visualization, Output formatting, and Output Saving at finer granularity.
- Benchmark scope: Unlike benchmarks focused on API calls, workflow descriptions, or code edits, ScienceAgentBench requires standalone programs adapted from real scientific publications.
3 EXPERIMENTAL SETUP
The experiments compare multiple LLMs and agent frameworks under repeated-task evaluation, including direct prompting, tool-based interaction, and iterative self-debugging. The setup also standardizes selection across runs while accommodating each generated program’s dependencies.
- Models and protocols: Experiments evaluate three open-weight LLMs, two proprietary LLMs, and OpenAI o1 using standardized prompting settings.
- Frameworks: Direct prompting generates a task program in one pass without interaction with a programming environment.
- Evaluation protocol: OpenAI o1 is evaluated only with direct prompting and self-debug because it was incompatible with OpenHands and did not allow hyperparameter changes as of 10/24/2024.
- Frameworks: OpenHands CodeAct equips agents with Python, bash, and web-browser tools for code generation and software engineering.
- Frameworks: Self-debug executes generated programs, exposes execution results, and iteratively improves programs through debugging turns.
- Evaluation protocol: Each task is repeated in three independent runs, with selection ordered by maximum SR, VER, CBS, and then minimum Cost.
4 RESULTS AND ANALYSIS
ScienceAgentBench results show that current agents achieve only low-to-moderate success on realistic scientific programming tasks, with self-debug, cost, expert knowledge, and task complexity materially affecting outcomes. Human ratings further identify data loading and processing as bottlenecks and complement execution-based metrics with fine-grained assessment.
- Main Results: 34.3% SR is the best performance achieved by Claude-3.5-Sonnet with self-debug and expert-provided knowledge across three attempts per task.The result underscores that current agents remain limited on realistic and challenging data-driven discovery tasks.
- Main Results: 16.7 →32.4 SR shows that self-debug nearly doubles Claude-3.5-Sonnet’s independent success rate without expert-provided knowledge.With expert knowledge, self-debug improves SR from 20.6 to 34.3 and VER from 41.2 to 86.3.
- Main Results: 10.8% more tasks are solved by Claude-3.5-Sonnet with self-debug than OpenHands, while API fees fall from $0.958 to $0.057.The comparison illustrates why both framework capability and cost matter in practical agent design.
- Main Results: Expert-provided knowledge consistently improves SR and CBS for most agents, but VER decreases for most agents.Knowledge can provide useful API names and task steps, yet specified tools may trigger incorrect or hallucinated API calls.
- Task Complexity and Failure Analysis: More than 75% of succeeded tasks have gold programs shorter than 58.6 lines, while many tasks with complex gold programs remain unsolved.For Bioinformatics and Computational Chemistry, failures concentrate on data processing and model development because the data are highly heterogeneous.
- Human Evaluation: 25% of failed programs receive ratings below 50 for data loading, while successful programs generally receive near-perfect ratings in that stage.Successful and failed programs form overlapping but distinguishable rating distributions, supporting fine-grained evaluation alongside outcome metrics.
5 CONCLUSION
ScienceAgentBench evaluates language agents on data-driven scientific discovery using tasks grounded in peer-reviewed research and expert validation. Its results support using agents to assist scientists while showing that current systems do not yet automate discovery tasks or full research pipelines.
- ScienceAgentBench compiles 102 real-world tasks from 44 peer-reviewed publications across four scientific disciplines, validated by nine subject matter experts.
- 34.3% is the highest reported task-solving rate, achieved by Claude-3.5-Sonnet with self-debug using expert-provided knowledge.
- Current language agents cannot yet automate data-driven discovery tasks or an entire research pipeline.
- The benchmark advocates using language agents to assist human scientists with tedious workflow tasks and assessing agents rigorously.
APPENDICES
The appendices document benchmark limitations, related work, task examples, construction details, evaluation analyses, case studies, expert validation, rubrics, prompts, and source repositories with licensing information.
- Appendix A covers limitations and future directions, while Appendix B presents related work.
- Appendix C provides example instructions for Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience tasks.
- Appendix D details benchmark construction, including annotated programs and success criteria.
- Appendix E reports additional main results, including performance statistics and error analyses for OpenHands CodeAct and self-debug.
- Appendix F presents case studies on OpenHands action spaces, expert-provided knowledge, and OpenAI o1 reasoning.
- Appendices G–I provide expert-validation materials, rubric examples, and prompt templates for direct prompting, self-debug, and OpenDevin.
- Appendix J lists source publications, repositories, licenses, and copyright information for benchmark components.
A LIMITATIONS AND FUTURE DIRECTIONS
The benchmark focuses on code generation for data-driven discovery, while recognizing that other agent capabilities and evaluation methods require further study. Its task and discipline coverage also reflects practical collection constraints.
- Scope: ScienceAgentBench evaluates language agents’ code generation for processing, analyzing, and visualizing scientific data.The benchmark targets code that is directly usable and verifiable by scientists.
- Scope: The benchmark does not assess literature summarization, idea suggestion, or experiment planning.The authors call for rigorous assessment of these capabilities separately.
- Coverage constraints: ScienceAgentBench uses Python programs, excludes tasks exceeding 10 minutes, and covers four disciplines because of annotation and collection constraints.These choices reduce coverage of other programming languages, large-scale data, complex methods, and additional disciplines.
- Benchmark context: Existing scientific-task benchmarks differ in output format, with DiscoveryBench-Real using abstract natural-language steps that are harder to evaluate rigorously.ScienceAgentBench instead focuses on code-generation tasks from real scientific workflows.
C EXAMPLE TASK INSTRUCTIONS
This section provides example task instructions for four scientific disciplines, grouped into two tables covering biology and chemistry, then geography and psychology.
- Bioinformatics and Computational Chemistry: Table C.1 provides example instructions for Bioinformatics and Computational Chemistry tasks.
- Disciplinary coverage: The examples span four disciplines represented in the benchmark’s task instructions.
- Geographical Information Science and Psychology & Cognitive Neuroscience: Table C.2 provides example instructions for Geographical Information Science and Psychology & Cognitive Neuroscience tasks.
D MORE DETAILS ABOUT BENCHMARK CONSTRUCTION
Benchmark programs are adapted from peer-reviewed publication repositories, validated by experts, and evaluated by whether generated programs reproduce annotated-program results.
- Program annotation: Annotated programs are extracted from open-source repositories of peer-reviewed publications rather than written by humans or generated by models.Annotators then remove redundant lines and load benchmark datasets before validation.
- Program annotation: Subject matter experts and other annotators validate the adapted programs.
- Evaluation criteria: Task success is determined by whether an LLM-generated program accurately reproduces the annotated program’s result.The criteria are task-specific and use validated publication-derived programs as references.
- Evaluation criteria: A multitask model on Clintox must achieve at least 0.77 ROC-AUC on the test set to satisfy that task’s success criterion.The threshold comes from five independent runs of the annotated program.
E.1 MEAN AND STANDARD DEVIATIONS OF AGENT PERFORMANCE
Agent performance is reported using the best of three independent runs, with mean performance and standard deviations provided as supplementary summaries.
- Reporting protocol: Results select the best of three independent runs for each task in all experiments.
- Reporting protocol: Mean performances and standard deviations are reported for agents evaluated without expert-provided knowledge.
- Reporting protocol: Mean performances and standard deviations are also reported for agents evaluated with expert-provided knowledge.
E.2 ERROR ANALYSIS OF OPENHANDS CODEACT AND SELF-DEBUG
The error analysis identifies three recurring weaknesses: executable programs can remain semantically wrong, environments can be misconfigured, and specialized editing commands can trigger loops and extra cost.
- Semantic correctness: 29/50 OpenHands CodeAct errors and 30/50 self-debug errors produced executable but semantically incorrect programs.Examples include simulating fake data or replacing a graph convolutional network with a simpler feed-forward network.
- Environment setup: 10/50 OpenHands CodeAct and 9/50 self-debug trajectories contained configuration errors when installing or configuring domain-specific tools.The problem affected both LLM-generated installation commands and human-developed packages.
- Tool interaction: 23/50 error trajectories involved Claude-3.5-Sonnet struggling with specialized OpenHands commands, especially for longer programs.Repeated command-generation loops wasted turns and increased API cost.
F.1 ACTION SPACE OF OPENHANDS
The OpenHands action space does not consistently improve agent performance: self-debug outperforms it for most evaluated LLMs, while specialized file-editing commands can hinder execution.
- Performance comparison: For four of five evaluated LLMs, self-debug performs better than OpenHands CodeAct; GPT-4o is the only exception.GPT-4o better leverages OpenHands tools, whereas Claude-3.5-Sonnet struggles with its specialized bash command for updating files.
- Specialized commands: Claude-3.5-Sonnet can spend repeated steps struggling with specialized OpenHands commands to edit program files correctly.Listing F.2 documents a failed edit example involving the specialized bash command.
- Browsing actions: GPT-4o’s OpenHands example uses browsing actions, including requests to a BrowsingAgent and visits to NeuroKit2 documentation.The listing records browser-based documentation lookup for the rsp_rrv function.
F.2 INFLUENCE OF EXPERT-PROVIDED KNOWLEDGE
Expert-provided knowledge can shift agents toward more specialized scientific tools, but it does not always improve metrics and can expose execution or contamination problems.
- Metric effects: Expert-provided knowledge does not always improve metrics because advanced tools may be unfamiliar and agents may generate executable but less meaningful programs.These two mechanisms are identified explicitly in the section’s analysis.
- Program specialization: Expert knowledge changed Claude-3.5-Sonnet from a basic random-forest approach to a DeepPurpose model using MPNN drug encoding and CNN target encoding.Without the knowledge, the agent used pandas and sklearn; with it, the generated program installed DeepPurpose and configured advanced encoders.
- Contamination checks: The expert-guided DeepPurpose program showed possible data contamination because its automatic data loader did not read the modified local dataset.The benchmark’s contamination-mitigation strategies captured this issue.
J PUBLICATIONS, REPOSITORIES, AND LICENSES
The appendix records the publications, repositories, and licenses used during benchmark data collection, including domain-specific publication lists and adapted repositories.
- Publications: The benchmark refers to publications in bioinformatics, computational chemistry, geographical information science, and psychology and cognitive neuroscience.These publication groups are listed in Tables J.1 and J.2.
- Repositories: 31 repositories were adapted during data collection, with their licenses documented in the appendix.The repository list and licensing information appear in Table J.3 and related license tables.
- Licenses: The appendix includes separate license tables for rasterio/rasterio and hackingmaterials/matminer.These entries are identified as Tables J.4 and J.5.