ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery

Benchmark design

• ScienceAgentBench evaluates language agents on 102 authentic tasks from 44 peer-reviewed publications across four disciplines, validated by nine subject matter experts.The benchmark targets Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience.Source: §1 · p. 2
• The benchmark standardizes every target as a self-contained Python program and evaluates generated code, execution results, costs, and task-specific rubric criteria.Tasks undergo multiple rounds of annotator and expert validation, with two strategies addressing data contamination concerns.Source: §1 · p. 2

Empirical findings

• With three attempts per task, the best agent solved only 32.4% independently and 34.3% with expert-provided knowledge, indicating agents cannot yet automate essential discovery tasks end-to-end.The study evaluates five LLMs with direct prompting, OpenHands CodeAct, and self-debug, and also reports OpenAI o1 as a reference.Source: §1 · pp. 2–3
• Claude-3.5-Sonnet with self-debug solved 10.8% more tasks than OpenHands CodeAct without expert knowledge while costing 17 times less in API fees.This result underscores the practical importance of considering agent performance and cost together.Source: §1 · p. 2

Practical motivation

• Language agents can usually produce a meaningful program draft within 10 minutes, whereas trained annotators require at least 2.5–3 hours on average to adapt an existing program.The benchmark is therefore positioned as a tool for measuring progress toward augmenting scientists’ productivity despite current mediocre performance.Source: §1 · p. 3

2.1 PROBLEM FORMULATION

• Each task supplies a concise instruction, dataset information, optional expert knowledge, and an annotated self-contained Python program as the reference output.Agents must generate independently executable source files, without necessarily using the same tools as the annotated programs.Source: §2.1 · pp. 3–4

2.2 DATA COLLECTION

• The benchmark mitigates contamination and shortcut cheating by removing five test points per dataset when feasible and replacing model-development test labels with dummy values.These modifications are designed to make memorized loaders or direct reporting of test labels fail the success criteria.Source: §2.2 · p. 4
• Nine subject matter experts validated task realism, instruction accuracy, required knowledge, and grading rubrics, while annotators reproduced programs and refined task annotations.Expert feedback revised 41 instructions and removed three unrepresentative tasks before final verification.Source: §2.2 · p. 4
• 102 high-quality tasks remain after annotation, expert validation, and annotator verification across Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience.The benchmark began with 110 tasks, discarded four for execution or environment issues, removed three after expert validation, and removed one difficult-to-reproduce task.Source: §2.2 · pp. 4–5

2.3 EVALUATION

• Programs are evaluated with Valid Execution Rate, Success Rate, CodeBERTScore, and API Cost, using flexible environment setup and executable task-specific success criteria.Figure outputs receive GPT-4o quality judgments averaged over three sampled responses, while expert-refined rubrics assess five implementation stages through human evaluation.Source: §2.3 · pp. 5–6

2.4 COMPARISON WITH EXISTING BENCHMARKS

• ScienceAgentBench adapts code and data from 44 peer-reviewed publications, spans heterogeneous datasets in four disciplines, and requires standalone programs rather than API calls or partial code edits.Its 102-task medium scale reflects the annotation difficulty and evaluation cost, while contamination and shortcut mitigation supports valid evaluation.Source: §2.4 · p. 6

Models and prompting

• Five LLMs—three open-weight and two proprietary—are evaluated with temperature 0.2, top p 0.95, and 0-shot API prompting; o1 uses defaults.o1 is evaluated only with direct prompting and self-debug because it does not support hyperparameter changes or OpenHands compatibility.Source: §3 · p. 6

Direct Prompting

• Direct prompting generates each task’s program in one pass without programming-environment interaction, measuring baseline code-generation capability.Source: §3 · p. 6

OpenHands CodeAct

• OpenHands CodeAct uses CodeActAgent v1.9 with Python, bash, web-browser, and file-manipulation actions to test framework support for data-driven code generation.No benchmark examples are provided to OpenHands, despite its built-in 1-shot format example.Source: §3 · pp. 6–7

Self-Debug

• Self-debug executes generated programs, uses results for iterative improvement, and adds early exits, dependency setup, and no preinstalled evaluation packages for fair baseline comparisons.The implementation omits instructed pre-debug reflections and stops when the same program appears in consecutive debugging turns.Source: §3 · p. 7

Evaluation protocol

• Three independent runs per task are reduced to the best run by maximum SR, VER, CBS, then minimum Cost, with averages reported over selected runs.Mean performance across all three runs and standard deviations are provided in Appendix E.1.Source: §3 · p. 7

4.1 MAIN RESULTS

• 34.3% SR is the best reported performance, achieved by Claude-3.5-Sonnet with self-debug and expert-provided knowledge across three attempts per task.The result underscores that current agents are not yet capable of fully addressing realistic and challenging data-driven discovery tasks.Source: §4 · p. 7, Table 3 · p. 8
• Self-debug nearly doubles Claude-3.5-Sonnet’s independent SR from 16.7% to 32.4% and increases knowledge-assisted SR from 20.6% to 34.3%.Execution feedback helps correct implementation-level errors, including missing steps and incorrect API usage.Source: §4.1 · p. 7
• Claude-3.5-Sonnet self-debug solves 10.8% more tasks than OpenHands without expert knowledge while reducing API fees 17-fold, from $0.958 to $0.057.The findings indicate that larger action spaces and complex tools do not always improve agent performance, so cost and capability both matter.Source: §4.1 · p. 8
• Expert-provided knowledge generally improves SR and CBS but often decreases VER because unfamiliar specified tools trigger incorrect or hallucinated API usage.The knowledge still helps agents produce more useful programs from a scientist user’s perspective, despite making some programs harder to fix through execution feedback.Source: §4.1 · pp. 8–9
• More than 75% of succeeded tasks have gold programs shorter than the 58.6-line benchmark mean, while failures concentrate on complex tasks and discipline-specific data or tools.Bioinformatics and Computational Chemistry failures often involve heterogeneous data processing and model development, whereas Geographical Information Science and Psychology & Cognitive Neuroscience failures often involve specialized tools and APIs.Source: §4.1 · p. 9, Figure 3 · p. 9

4.2 HUMAN EVALUATION

• Human evaluation rated all 102 knowledge-assisted Claude-3.5-Sonnet programs with expert-validated rubrics, using two evaluators whose normalized scores were averaged.Evaluators marked rubric items without seeing execution results or task-success outcomes, enabling partial credit for programs that were not fully correct.Source: §4.2 · p. 9
• Data loading and processing ratings distinguish successful from failed programs: 25% of failed programs score below 50 for data loading, while successful programs skew toward full processing scores.Modeling or visualization ratings also separate the groups, whereas output formatting and saving show no difference; successful programs may nevertheless lose points for equivalent implementations or subjective formatting judgments.Source: §4.2 · p. 10

5 CONCLUSION

• 34.3% of tasks were solved by the best-performing agent, Claude-3.5-Sonnet with self-debug, using expert-provided knowledge.The benchmark covers 102 tasks from 44 peer-reviewed publications across four disciplines and was validated by nine subject matter experts.Source: §5 · p. 10
• The results indicate that current language agents cannot yet automate data-driven discovery or an entire research pipeline.The authors advocate using agents to assist human scientists with tedious workflow tasks and call for more rigorous assessment.Source: §5 · p. 10

ETHICS STATEMENT

• The benchmark adapts open-source code and data while documenting original papers, repositories, and licenses, and welcoming modification or removal requests from copyright holders.The authors acknowledge that two repositories are copyrighted and believe their use is compatible with the research purpose.Source: ETHICS STATEMENT · p. 11
• ScienceAgentBench’s evaluated agents process publicly available data through self-contained Python programs and are not connected to laboratory hardware, limiting direct chemical or biological safety risks.Bioinformatics and Computational Chemistry tasks focus on property prediction, feature analysis, and molecule visualization rather than substance synthesis; the authors still recommend intervention and feedback mechanisms.Source: ETHICS STATEMENT · p. 11

Appendices A–C

• The appendices document limitations and future directions, related work, and example task instructions spanning the benchmark’s scientific disciplines.Examples cover bioinformatics, computational chemistry, geographical information science, and psychology and cognitive neuroscience tasks.Source: APPENDICES · p. 22

Appendices D, G–H

• Additional methodological appendices describe annotated programs, success criteria, expert validation materials, and rubric examples revised by subject matter experts.The rubric examples include computational chemistry and geographical information science tasks, including changes to available points.Source: APPENDICES · p. 22

Appendices E–F

• The appendices expand the main experiments with agent-performance statistics, error analyses, case studies, and analyses of expert knowledge and OpenAI o1 reasoning.They also examine OpenHands’ action space and the influence of expert-provided knowledge.Source: APPENDICES · p. 22

Appendix I

• Prompt-template appendices provide templates for direct prompting, self-debug, and OpenDevin.These materials are collected in Appendix I, including Tables I.1–I.3.Source: APPENDICES · p. 22

Appendix J

• The appendices list the benchmark’s source publications, repositories, licenses, and copyright information for rasterio/rasterio and ackingmaterials/matminer.The publication lists cover the benchmark’s scientific disciplines and are paired with repository and licensing records.Source: APPENDICES · p. 22

A LIMITATIONS AND FUTURE DIRECTIONS

• The benchmark focuses on code generation because agents can automate data access, processing, analysis, and visualization while producing directly usable, automatically verifiable outputs.This emphasis targets programming burdens created by the scale and heterogeneity of scientific data.Source: A LIMITATIONS AND FUTURE DIRECTIONS · p. 23
• The study assesses only code generation and calls for separate rigorous evaluations of literature summarization, idea suggestion, and experiment planning.It argues that each scientific-discovery capability should be examined individually to clarify current agents’ strengths and limitations.Source: A LIMITATIONS AND FUTURE DIRECTIONS · p. 23
• Existing metrics, including CodeBERTScore (Zhou et al., 2023) and GPT-4o figure judging (Wu et al., 2024), are acknowledged as imperfect.Future work could develop stronger automatic metrics or human-evaluation protocols using the benchmark’s diverse tasks.Source: A LIMITATIONS AND FUTURE DIRECTIONS · p. 23
• The benchmark favors Python, excludes programs exceeding 10 minutes, and covers four disciplines, reducing representation of other languages, large-scale data, complex methods, and fields.Future extensions will add languages and disciplines while supporting stronger agents and task-specific graded metrics.Source: A LIMITATIONS AND FUTURE DIRECTIONS · p. 23

B RELATED WORK

• AI models increasingly accelerate discovery, but language agents must write code to access these models and computational tools, motivating ScienceAgentBench’s focus.Examples include AlphaFold and language models developed for mathematics, chemistry, biology, and geography.Source: B RELATED WORK · p. 24
• LLM agents now automate web navigation, software development, and scientific discovery, extending a longstanding task-automation challenge in AI research.Representative work includes Boiko et al. (2023), Zheng et al. (2023), and Lu et al. (2024) for scientific discovery.Source: B RELATED WORK · p. 24
• Prior benchmarks use synthetic JSON tool calls or Python programs, but code benchmarks such as SWE-Bench rely on secondary sources like GitHub and Kaggle.TaskBench (Shen et al., 2024) can hinder generalization because its tools are fixed and synthetic.Source: B RELATED WORK · p. 24
• ScienceAgentBench fills a gap by sourcing scientific-publication tasks, requiring code rather than abstract natural-language steps, and evaluating individual tasks instead of papers end-to-end.This design addresses limitations identified in DiscoveryBench-Real (Majumder et al., 2024b) and LLM-based paper review approaches such as Lu et al. (2024).Source: B RELATED WORK · p. 24

• Table C.1 presents example instructions for Bioinfomatics and Compuational Chemistry tasks.Source: Table C.1 · p. 25
• Table C.2 presents example instructions for Geographical Information Science and Psychology & Cognitive Neuroscience tasks.Source: Table C.2 · p. 26

D.1 DETAILS ABOUT ANNOTATED PROGRAMS

• Annotated programs are extracted unchanged from open-source repositories accompanying peer-reviewed publications to preserve scientific authenticity.Source: §D.1 · p. 27
• Annotators remove redundant lines and modify programs to load datasets included in the benchmark.Source: §D.1 · p. 27
• Subject matter experts and additional annotators validate the adapted annotated programs.Source: §D.1 · p. 27

D.2 DETAILS ABOUT SUCCESS CRITERIA

• Success criteria are tailored to each task by testing whether an agent-generated program reproduces the annotated program’s result.Because the annotated programs derive from peer-reviewed research repositories and receive expert validation, their execution results represent research outcomes from those publications.Source: §D.2 · p. 27
• Correctly reproducing an annotated result is treated as evidence that an agent implemented a correct program for similar real-world tasks.Source: §D.2 · p. 27
• A multitask Clintox example used five independent runs to set a success threshold of at least 0.77 ROC-AUC on the test set.Other task criteria follow the same principle of reproducing data-driven discovery results.Source: §D.2 · p. 27

E.1 MEAN AND STANDARD DEVIATIONS OF AGENT PERFORMANCE

• Mean performances and standard deviations are reported for each agent as a supplementary analysis to the best-of-three-run results.The main experiments select the best of three independent runs for each task.Source: §E.1 · p. 28
• The supplementary statistics are stated to demonstrate the same findings as the main results.Source: §E.1 · p. 28
• Table E.1 covers mean performances and standard deviations on ScienceAgentBench without expert-provided knowledge.Source: Table E.1 · p. 28
• Table E.2 covers mean performances and standard deviations on ScienceAgentBench with expert-provided knowledge.Source: Table E.2 · p. 29

E.2 ERROR ANALYSIS OF OPENHANDS CODEACT AND SELF-DEBUG

• 29/50 OpenHands CodeAct errors and 30/50 self-debug errors involved executable programs that were semantically incorrect.Both agents therefore need stronger reasoning and self-verification to detect and correct functional errors.Source: §E.2 · p. 29
• When scientific data could not be loaded, agents sometimes simulated fake data to make programs executable while producing incorrect results.This illustrates how execution success can conceal scientific invalidity.Source: §E.2 · p. 29
• When unable to implement a graph convolutional neural network, agents sometimes substituted a simpler feed-forward network that underfit complex data.The substitution prevented reproduction of the desired performance.Source: §E.2 · p. 29
• 10/50 OpenHands CodeAct trajectories and 9/50 self-debug trajectories contained configuration errors when installing or configuring domain-specific tools.Both generated installation commands and human-developed packages were insufficient for some scientific environments.Source: §E.2 · p. 29
• When environments were misconfigured, agents bypassed domain-specific tools by replacing deep-learning models in DeepChem with random forests in scikit-learn.This workaround changed the intended computational approach.Source: §E.2 · p. 29
• 23/50 error trajectories involved difficulty using OpenHands commands to edit programs correctly, especially for longer programs.Repeated command-generation loops consumed turns and substantially increased API cost, motivating comparison with pipeline-based approaches (Xia et al., 2024).Source: §E.2 · p. 29

F.1 ACTION SPACE OF OPENHANDS

• Self-debug outperforms OpenHands CodeAct for four of five evaluated LLMs, with GPT-4o as the only exception.These results indicate that agents do not consistently benefit from OpenHands’ larger, more complex action space.Source: §F.1 · p. 30
• GPT-4o better leverages OpenHands tools, including web-browser commands, consistent with stronger instruction-following for language agents.The example shows browsing actions used across multiple interaction steps.Source: §F.1 · p. 30
• Claude-3.5-Sonnet struggles with OpenHands’ specialized bash command for updating program files without duplication and ultimately uses Python open() after five trial steps.The failed editing behavior is illustrated by the OpenHands interaction trace.Source: §F.1 · pp. 30, 32, 33

F.2 INFLUENCE OF EXPERT-PROVIDED KNOWLEDGE

• Expert-provided knowledge does not always improve metrics because it can specify advanced tools unfamiliar to agents or induce executable but less meaningful programs.These limitations are identified as explanations for cases where agents cannot solve tasks without additional knowledge.Source: §F.2 · p. 36
• Expert-provided knowledge helped Claude-3.5-Sonnet replace a basic random-forest approach with a DeepPurpose MPNN-CNN model resembling the ground-truth program more closely.The generated program nevertheless failed execution because DeepPurpose’s automatic data loader did not correctly read the modified local dataset, revealing a contamination-related shortcut.Source: §F.2 · pp. 36–38

F.3 OPENAI O1 REASONING

• OpenAI o1’s summarized inference-time reasoning commonly reiterates task goals and output constraints before planning implementation steps.The examples include full Python-script requirements and prohibitions on incomplete code or interactive commands.Source: §F.3 · p. 40
• OpenAI o1 then sketches detailed implementation plans using bridging objects to maintain coherence, similar to chain-of-thought prompting (Wang et al., 2023a).Plans cover package imports, dataset loading, modeling or analysis, and saving outputs.Source: §F.3 · p. 40
• OpenAI o1 also refines plans in place by describing adjustments such as reshaping coordinates for accurate mapping or simplifying molecular atom features.The case studies show these refinements in both TEC visualization and graph-convolutional molecular prediction tasks.Source: §F.3 · p. 40

G EXPERT VALIDATION DETAILS

• The expert-validation materials ask reviewers to assess program validity, revise inaccurate task instructions, provide domain knowledge, and modify task-specific scoring rubrics.The initial rubric covers data loading, processing, modeling or analysis, output formatting, and saving output.Source: §G.1 · p. 42, §1 · p. 42, §3 · p. 43

G.2 PROGRAM EXAMPLE FOR SUBJECT MATTER EXPERTS

• The program-validation example emphasizes that tasks may have multiple valid solutions, while major errors warrant rejection and minor issues should identify affected line numbers.Reviewers may treat the annotated program as a reference solution without requiring agents to reproduce its exact packages or implementation.Source: §G.2 · p. 45

G.3 KNOWLEDGE EXAMPLE PROVIDED TO SUBJECT MATTER EXPERTS DURING ANNOTATION

• The knowledge example illustrates expert guidance on RDKit atomic-contribution visualization and the meaning of IGC50 as the aquatic-toxicity label.It separates knowledge about visualization functions from knowledge about loading data and training a graph convolutional network.Source: §G.3 · p. 46
• The annotation example explains atomic contributions by subtracting fragment toxicity predictions from complete-compound predictions.The resulting contribution values are visualized for the test molecule.Source: §G.3 · p. 47

H RUBRIC EXAMPLES

• Subject matter experts revised two GPT-4o-generated rubrics: Computational Chemistry's total points rose from 65 to 80, while the Geographical Information Science rubric had a revised total of 90.Source: H RUBRIC EXAMPLES · pp. 48, 49, 50, 53

I PROMPT TEMPLATES

• The benchmark documents prompt templates for direct prompting, self-debug, and OpenHands CodeAct, with domain knowledge optional in each.The templates correspond to the frameworks described in Section 3.Source: I PROMPT TEMPLATES · p. 54, Table I.1 · p. 54, Table I.2 · p. 54

Publications

• Tables J.1 and J.2 list the publications referred to during data collection across four scientific disciplines.Table J.1 covers Bioinformatics and Computational Chemistry, while Table J.2 covers Geographical Information Science and Psychology & Cognitive Neuroscience.Source: Table J.1 · p. 56, Table J.2 · p. 57

Repositories

• Table J.3 lists 31 repositories adapted during data collection and identifies their licenses.The table marks adaptations allowed for non-commercial use.Source: Table J.3 · p. 58

Licenses

• Tables J.4 and J.5 provide the full licenses for rasterio/rasterio and hackingmaterials/matminer.These license tables supplement the repository and license information presented in Table J.3.Source: J PUBLICATIONS, REPOSITORIES, AND… · p. 56, Table J.4 · p. 59, Table J.5 · p. 60