PaperlayerView exact revision on arXiv

Source-linked AI summary

CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, Arvind Narayanan

arXiv:2409.11363v2cs.CLcs.AIcs.MA

TL;DR

Even with code and data, reproducing published results can be difficult, while assessing reproducibility requires expertise. CORE-Bench evaluates agents on tasks derived from scientific papers; task-specific modifications help, but the best baseline achieves only 21% test-set accuracy. The authors present reproducibility as a necessary step toward agents that conduct novel research.

  • Problem

    Reproducing scientific results remains challenging despite available code and data, and verifying reproducibility requires expertise.

    Source: §1 · p. 2

  • Method

    CORE-Bench evaluates agents on 270 tasks derived from 90 papers, across three difficulty levels, using code and data from reproducible repositories.

    Source: §1 · p. 3, §2.1 · p. 5

  • Results

    Task-specific modifications improve accuracy, but the best baseline achieves 21% test-set accuracy.

    Source: §5 · p. 13

  • Takeaways & Limitations

    The authors describe reproducing existing scientific work as a necessary, more attainable step toward agents that conduct novel research.

    Source: §2.2 · p. 7

    CORE-Bench tests reproduction of code-associated results from reproducible papers, not whether the results reported in those papers are correct.

    Source: §2.1 · p. 6

Abstract

from arXiv · show

AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research. To spur the development of useful agents, we need benchmarks that are challenging, but more crucially, directly correspond to real-world tasks of interest. This paper introduces such a benchmark, designed to measure the accuracy of AI agents in tackling a crucial yet surprisingly challenging aspect of scientific research: computational reproducibility. This task, fundamental to the scientific process, involves reproducing the results of a study using the provided code and data. We introduce CORE-Bench (Computational Reproducibility Agent Benchmark), a benchmark consisting of 270 tasks based on 90 scientific papers across three disciplines (computer science, social science, and medicine). Tasks in CORE-Bench consist of three difficulty levels and include both language-only and vision-language tasks. We provide an evaluation system to measure the accuracy of agents in a fast and parallelizable way, saving days of evaluation time for each run compared to a sequential implementation. We evaluated two baseline agents: the general-purpose AutoGPT and a task-specific agent called CORE-Agent. We tested both variants using two underlying language models: GPT-4o and GPT-4o-mini. The best agent achieved an accuracy of 21% on the hardest task, showing the vast scope for improvement in automating routine scientific tasks. Having agents that can reproduce existing work is a necessary step towards building agents that can conduct novel research and could verify and improve the performance of other research agents. We hope that CORE-Bench can improve the state of reproducibility and spur the development of future research agents.

1 Introduction

Computational reproducibility remains difficult even when code and data are available, and the paper asks whether AI agents can automate it. CORE-Bench evaluates that ability; the best baseline achieves 21% accuracy on the hardest task.

Motivation

  • Even when studies provide code and data, reproducibility can be impeded by unspecified library versions, hardware or operating-system differences, incompatibilities, and stochastic results.The paper surveys evidence of irreproducibility across fields.Source: §1 · p. 2, Table 1 · p. 2

CORE-Bench

  • CORE-Bench comprises 270 tasks from 90 papers across computer science, social science, and medicine, with three difficulty levels and Python or R codebases.Tasks may require coding, shell interaction, retrieval, tool use, and figure interpretation.Source: §1 · p. 3

Potential utility

  • The authors argue that successful agents could help authors, independent researchers, and editors assess computational reproducibility.The paper presents reproducibility as a necessary step toward agents that conduct novel research.Source: §1 · pp. 3–4

Evaluation results on baseline agents

  • 21% accuracy on the hardest task shows substantial room for improvement, despite task-specific modifications that improve general-purpose agents.The task-specific agent reached 60% accuracy on the easiest task.Source: §1 · p. 4
  • The evaluation harness runs tasks in parallel virtual machines, reducing evaluation time from over 20 days to hours while separating benchmark and agent code.The harness also standardizes hardware access and prevents agents from tampering with the benchmark.Source: §1 · p. 4

2 CORE-Bench: Evaluating agents on computational reproducibility

CORE-Bench targets the research-reproduction process, an area the authors say existing benchmarks have not addressed. It uses selected CodeOcean capsules to create tasks that test distinct skills and approximate real-world reproduction.

2 CORE-Bench: Evaluating agents on computational reproducibility

  • CORE-Bench targets automating research reproduction, a part of the research pipeline that the authors say has not yet received attention from existing benchmarks.The paper distinguishes reproducing released work from conducting new research.Source: §2 · p. 4

2.1 Benchmark Construction

  • The benchmark selects 90 reproducible CodeOcean papers, splits them into 45 training and 45 testing papers, and creates task questions about successfully reproduced outputs.Tasks are marked correct only when all questions are answered correctly, and each includes at least one question that cannot be solved by guessing.Source: §2.1 · pp. 5–6
  • The 270 tasks span three difficulty levels that vary the reproduction information provided, from focused skill tests to a setup resembling real-world paper reproduction.The first two levels can assess skills such as information retrieval or terminal interaction; skills expand as difficulty increases.Source: Table 3 · p. 6

2.2 Why use CORE-Bench?

  • Tasks combine text and image outputs, testing skills such as instruction-following, debugging, retrieval, and interpreting results across disciplines.Questions may require reading command-line output or extracting information from figures, plots, PDFs, or tables.Source: §2.2 · p. 6
  • The authors frame reproducibility as a necessary, more attainable first step toward agents that can conduct novel scientific research.They also argue that CORE-Bench tasks closely correspond to researchers’ real-world work.Source: §2.2 · p. 7

3 Baseline agents and evaluation setup

The section contrasts general-purpose AutoGPT with level-tailored CORE-Agent and describes the isolated, parallel virtual-machine harness used to evaluate them.

3 Baseline agents and evaluation setup

  • AutoGPT gained a query_vision_language_model tool for analyzing figures and plots.The tool takes an image and query and returns the vision-language model’s response.Source: §3 · p. 7
  • CORE-Agent added difficulty-specific prompts and an output-file check to AutoGPT, with the prompts tailored to each benchmark level.Table 4 summarizes the level-specific hints; the report-file check applies across all levels.Source: §3 · pp. 7–8, Table 4 · p. 8
  • Both agent variants used GPT-4o and GPT-4o-mini, and runs stopped when API costs exceeded $4 per task.The models were selected to examine the cost-accuracy trade-off.Source: §3 · p. 8
  • Task accuracy counted a task as correct only when every task question was answered correctly; average cost measured API costs across requests.Source: §3 · p. 9
  • Over 20 days sequentially versus a little over two hours with the harness for 270 tasks and 181 questions.Tasks ran in isolated VMs, with results evaluated locally after completion.Source: §3 · p. 9

4 Results

CORE-Agent with GPT-4o led all three difficulty levels, but its accuracy fell to 21.48% on CORE-Bench-Hard. Performance varied with task difficulty, model adaptation, question format, programming language, and agents’ ability to retrieve outputs and install dependencies.

4 Results

  • CORE-Agent with GPT-4o solved 60.00% of CORE-Bench-Easy, 57.78% of CORE-Bench-Medium, and 21.48% of CORE-Bench-Hard tasks.Table 5 reports average accuracy across three runs for CORE-Agent; AutoGPT was run once due to cost constraints.Source: §4 · p. 9, Table 5 · p. 10

4.1 Accuracy varies by difficulty level as expected

  • CORE-Agent with GPT-4o-mini scored 44.44%, 32.59%, and 16.30% on Easy, Medium, and Hard, respectively, as tasks required progressively more reproduction work.Easy supplies code outputs, Medium supplies a Docker command, and Hard requires installing dependencies and determining the reproduction command.Source: §4.1 · p. 10

4.2 Task specific modifications improve accuracy, especially for weaker models

  • On CORE-Bench-Easy, a few prompt modifications and a programmatic output-format check raised AutoGPT performance from 35.6% to 60.60% with GPT-4o and from 8.9% to 44.44% with GPT-4o-mini.The authors hypothesize that agents using stronger future models will require even fewer task-specific modifications.Source: §4.2 · p. 10

4.3 Stronger models lead to higher accuracy despite a lower token budget

  • GPT-4o outperformed GPT-4o-mini on both agents despite its higher token cost, while GPT-4o-mini agents tended to cost 3–5x less per task.With a $10 limit on the Hard training set, GPT-4o rose from 26% to 31% and GPT-4o-mini remained unchanged; successful GPT-4o CORE-Agent tasks averaged $0.54 versus $2.59 for failed tasks.Source: §4.3 · pp. 10–11, Figure 6 · p. 11, Figure 7 · p. 11

4.4 Written questions are easier than vision questions

  • On Easy test tasks, GPT-4o CORE-Agent answered 87.88% of written questions and 59.26% of vision questions correctly; GPT-4o-mini answered 81.81% and 37.78%, respectively.Vision tasks often require finding and interpreting figures, while written answers are often directly available in terminal output.Source: §4.4 · p. 11, Figure 8 · p. 12

4.5 Python tasks are much easier than R

  • Agents performed much better on Python than R tasks, where outputs could require reading full PDF manuscripts and package dependencies could take longer to install.Computer Science tasks were disproportionately in Python, which might explain why that discipline tended to be the most reproducible.Source: §4.5 · p. 12, Figure 8 · p. 12

4.6 Agents struggle to retrieve results from many files and often time out while installing dependencies

  • Agents struggled to identify the correct figure among multiple output files, and Hard-task agents often stalled while repeatedly resolving dependency versions until reaching the cost limit.On Medium, AutoGPT could follow competing README instructions instead of the Docker command; CORE-Agent tended not to struggle with that because of its task-specific instructions.Source: §4.6 · p. 12

4.7 Better guardrails are needed to deploy safe agents

  • After an agent attempted to create a CodeOcean account, the authors restricted the evaluation harness from accessing that domain and called for stronger safeguards against arbitrary web actions.The paper notes that JavaScript-rendering limits prevented most damaging actions in these experiments, but recommends additional safety checks as agents advance.Source: §4.7 · pp. 12–13

5 Conclusion

Reproducing released research is a prerequisite to building on existing knowledge, yet assessing reproducibility is labor-intensive. CORE-Bench aims to spur agents that reduce this burden, while baseline results show substantial room for improvement.

5 Conclusion

  • Reproducing released research is described as a prerequisite for building on existing knowledge.Source: §5 · p. 13
  • The authors hope CORE-Bench will stimulate agents that reduce the time and effort required for routine reproducibility work.Assessing reproducibility can require substantial human labor to read, understand, and run code.Source: §5 · p. 13
  • 21% test-set accuracy was the best baseline agent’s result, showing substantial room to improve computational reproducibility automation.The paper reports this as the best baseline agent’s accuracy on CORE-Bench test tasks.Source: §5 · p. 13

A.1 Original CodeOcean Dataset

The authors assembled a CodeOcean capsule dataset, filtered it using ten criteria, and illustrated how capsule run files were assessed.

A.1 Original CodeOcean Dataset

  • The authors scraped metadata for 5,090 CodeOcean capsules, manually exported their environment files, and filtered the dataset using ten criteria.Source: §A.1 · p. 18

A.2 Examples of capsule selection criteria

  • Table 2 lists ten filtering criteria, illustrated with an accepted capsule run file and an output from a rejected capsule.Source: §A.2 · p. 18
  • The accepted example satisfies criterion six: its code can be reproduced with a relatively simple Bash command.Source: §A.2 · p. 18

A.3 Task question construction

Task questions were built from outputs of successfully reproduced capsules, with questions directing agents to extract specified values or labels. The authors used several ways to identify figures or tables and report the benchmark’s capsule breakdowns in supplementary tables.

A.3 Task question construction

  • 90 capsules yielded 181 manually written task questions, with one to eight questions per capsule based on outputs from successful runs.Requested outputs could include a model’s accuracy, a figure’s axis label, or another relevant metric.Source: §A.3 · p. 18
  • Questions identified tables or figures by measured metric, title, or figure number from a results filename or document.Source: §A.3 · pp. 18–19
  • Supplementary tables report capsule counts by discipline and task modality, and by discipline across train and test sets.Source: Table A1 · p. 19, Table A2 · p. 19

A.4 Breakdown of task questions by discipline and modality

CORE-Bench includes a similar number of vision- and language-based questions, but more computer science capsules because fewer qualifying social science and medical capsules were available. The harness runs agents in isolated virtual machines, and the paper reports training-set comparisons and test-set accuracy and cost summaries.

A.4 Breakdown of task questions by discipline and modality

  • Computer science capsules outnumber the other disciplines because fewer social science and medical capsules met the selection criteria; total vision- and language-based question counts are similar.Source: §A.4 · p. 19

B Harness Details

  • The harness creates a virtual machine for each task-agent pair, transfers capsule and agent files, runs the agent, and retrieves results after task completion is logged.Source: B Harness Details · p. 19
  • On the train set, task-specific agents consistently outperformed generalist agents.Figure A1 compares AutoGPT and CORE-Agent with GPT-4o and GPT-4o-mini.Source: Figure A1 · p. 20
  • Table A3 reports test-set accuracy and costs by task difficulty across three trials, with 95% confidence intervals.Source: Table A3 · p. 20

C Experimental Details

On the training set, CORE-Agent consistently outperformed AutoGPT, and GPT-4o outperformed GPT-4o-mini.

C.1 Agent Accuracy on the Train Set

  • The reported comparison concerns agent accuracy on the training set.Source: §C.1 · p. 20
  • CORE-Agent consistently outperformed AutoGPT on the training set.The authors plotted both agents’ train-set accuracy in Figure A1.Source: §C.1 · p. 20
  • GPT-4o outperformed GPT-4o-mini on the training set.Source: §C.1 · p. 20

C.2 Confidence Intervals on Test Set

The test-set confidence intervals for the top-performing agent were under five percentage points across difficulty levels, while pass@k results rose with retries on CORE-Bench-Hard.

C.2 Confidence Intervals on Test Set

  • The top-performing agent’s accuracy confidence interval was under 5 percentage points on every difficulty level.Experiments were run three times to estimate 95% confidence intervals over mean accuracy and cost.Source: §C.2 · p. 20
  • GPT-4o-mini had larger accuracy confidence intervals than GPT-4o, which the authors interpret as lower reliability.Source: §C.2 · p. 20

C.3 Pass@k on the Test Set

  • 22.2% pass@1 and 31.1% pass@3 for CORE-Agent with GPT-4o; 15.6% pass@1 and 26.7% pass@3 with GPT-4o-mini on CORE-Bench-Hard.Source: §C.3 · p. 20

C.4 Pass∧k on the Test Set

Pass∧k results show that CORE-Agent’s chance of consistently solving the same hard tasks fell across repeated trials. The appendix also details agent modifications, prompts, and recurring failure cases.

C.4 Pass∧k on the Test Set

  • 22.22% pass∧1 and 8.89% pass∧3 for CORE-Agent with GPT-4o; 15.56% pass∧1 and 6.67% pass∧3 with GPT-4o-mini on CORE-Bench-Hard.The pass∧k metric is the chance that all k task trials succeed.Source: §C.4 · pp. 20–21, Figure A3 · p. 21

D.1 AutoGPT Bug Fixes and Changes

  • AutoGPT and CORE-Agent received general-use fixes for long tool outputs and shell commands, alongside CORE-Agent’s task-specific prompts.Long outputs were truncated to preserve their beginnings and endings; enabling shell=True allowed shell-specific commands such as &&.Source: §D.1 · p. 21

D.2 CORE-Agent Prompts

D.3.1 Using incorrect figure during information extraction (CORE-Bench-Easy)

  • On an easy task, CORE-Agent read the wrong figures and returned an incorrect p-value despite the relevant figure being available in the results.Source: §D.3.1 · p. 24

D.3.2 Manually reproducing the code instead of using Docker (CORE-Bench-Medium)

  • On a medium task, CORE-Agent with GPT-4o-mini manually reproduced code despite Docker instructions, eventually hitting the context limit and failing.Source: §D.3.2 · p. 25

D.3.3 Being unable to install the correct version of dependencies (CORE-Bench-Hard)

  • On a hard task, CORE-Agent installed network-diffusion 0.14.4, but the required MultiSpreading import was supported only in version 0.6.The authors note that selecting a library version without documentation can be difficult without external knowledge.Source: §D.3.3 · p. 26

D.3.4 Attempting to look up the capsule on CodeOcean (CORE-Bench-Hard)

  • After failing to find requirements.txt, CORE-Agent’s CodeOcean lookup was blocked because the capsule page required JavaScript unavailable through its browsing capabilities.Source: §D.3.4 · pp. 27–28

E Reproducibility Study Details

The reproducibility-study appendix explains how the authors compared prior reports and classified outcomes from the 2022 Machine Learning Reproducibility Challenge.

E Reproducibility Study Details

  • The authors reported reproduced-study counts according to the format used in each paper’s results.Source: E Reproducibility Study Details · p. 29
  • For papers reporting reproducibility percentages, the authors manually calculated and rounded the number of reproducible studies.This applied to Stockemer et al. (2018), Gertler et al. (2018), Collberg & Proebsting (2016), Hardwicke et al. (2021), and Raff (2019).Source: E Reproducibility Study Details · p. 29
  • Some cited studies reported reproducible results or metrics rather than counts of reproduced papers; McCullough et al. (2006) gave only an approximate paper count.The metrics-based reports included Gilbert et al. (2012), Trisovic et al. (2022), Samuel & Mietchen (2024), Pérignon et al. (2024), and Belz et al. (2021).Source: E Reproducibility Study Details · p. 29
  • Of 28 challenge papers with both data and code, 10 were partially reproduced; full reproduction required all main claims to hold, while significant result deviations counted as errors.Additional experiments outside the original paper were excluded; many codebases also had errors, outdated packages, or limited documentation.Source: E Reproducibility Study Details · pp. 29–30

Related papers

Loading 2409.11363v2…