PaperlayerView exact revision on arXiv

Source-linked AI summary

EXP-Bench: Can AI Conduct AI Research Experiments?

Patrick Tser Jern Kon, Jiachen Liu, Xinyi Zhu, Qiuyi Ding, Jingjia Peng, Jiarong Xing, Yibo Huang, Yiming Qiu, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury, Matei Zaharia, Ang Chen

arXiv:2505.24785v2cs.AI

TL;DR

AI agents struggle with rigorous, end-to-end experimentation, and research details are fragmented across papers and codebases. EXP-Bench couples influential AI papers with their code and uses a semi-automated pipeline to build experiment tasks. Agents generally score below 30% across metrics, with OH+o3-mini scoring 1.4% under the strict All✓ metric.

  • Problem

    Agents struggle with rigorous, end-to-end experimentation, while research details are fragmented across papers and codebases, making high-fidelity benchmark construction difficult to scale.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    EXP-Bench couples influential AI research papers with their codebases and uses a semi-automated pipeline with lightweight human verification to create end-to-end experiment tasks.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Agents generally scored below 30% across metrics, except several OH models reached ≈41% in RL implementation; under All✓, OH+o3-mini scored 1.4%.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    EXP-Bench serves as an evaluation tool and dataset to guide future AI agents in conducting end-to-end AI research experimentation.

    Source: §6 · p. 9

    EXP-Bench focuses on experimentation and does not fully capture literature review, unstructured research ideation, or the iterative and unpredictable path of real-world discovery.

    Source: §5 · p. 8

Abstract

from arXiv · show

Automating AI research holds immense potential for accelerating scientific progress, yet current AI agents struggle with the complexities of rigorous, end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed to systematically evaluate AI agents on complete research experiments sourced from influential AI publications. Given a research question and incomplete starter code, EXP-Bench challenges AI agents to formulate hypotheses, design and implement experimental procedures, execute them, and analyze results. To enable the creation of such intricate and authentic tasks with high-fidelity, we design a semi-autonomous pipeline to extract and structure crucial experimental details from these research papers and their associated open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks from 51 top-tier AI research papers. Evaluations of leading LLM-based agents, such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial capabilities: while scores on individual experimental aspects such as design or implementation correctness occasionally reach 20-35%, the success rate for complete, executable experiments was a mere 0.5%. By identifying these bottlenecks and providing realistic step-by-step experiment procedures, EXP-Bench serves as a vital tool for future AI agents to improve their ability to conduct AI research experiments. EXP-Bench is open-sourced at https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench.

1 Introduction

EXP-Bench evaluates whether AI agents can carry out rigorous, end-to-end research experiments, addressing a gap left by their emerging abilities in isolated research tasks. Its semi-automated pipeline builds tasks from published research and code, and agent evaluations reveal substantial difficulties across experimental design, implementation, and execution.

1 Introduction

  • EXP-Bench tests agents on complete experiments from influential peer-reviewed AI papers and their code, from formulating hypotheses through deriving conclusions.Agents receive a research question, high-level method, and starter code, then design, implement, and execute procedures.Source: Figure 1 · p. 1, §1 · p. 2
  • High-fidelity task curation is difficult because experimental details and conditions are often omitted from papers or scattered across papers, supplements, and codebases.Accurate interpretation therefore requires domain expertise and makes manual curation labor-intensive and difficult to scale.Source: §1 · p. 2
  • The semi-automated pipeline extracts task details from papers, supplementary materials, and code, locates implementation scripts, and uses execution-based validation.The resulting benchmark contains 461 tasks and 12,737 individually gradable subtasks from 51 NeurIPS and ICLR 2024 papers.Source: §1 · p. 2
  • 39.7% of failures involved missing essential implementation components, while 16.1% involved misclassified design variables and execution errors included 29.4% environment or dependency misconfigurations and 23.8% script-level errors.The evaluation covers design, implementation, execution, and conclusion; the authors identify these bottlenecks as targets for improving agents.Source: §1 · p. 2

2 Related work

Prior benchmarks assess scientific reasoning, coding, or machine-learning tasks, but often isolate components or simplify the research setting. EXP-Bench is positioned as an evaluation of holistic, step-by-step AI research experimentation.

2 Related work

  • EXP-Bench targets end-to-end, step-by-step research experimentation, whereas prior scientific reasoning and coding benchmarks often assess static artifacts, code snippets, or post-hoc analysis.BoxingGym explores simulated theory formation; AAAR and Lab-Bench assess reasoning or experimental design from static artifacts, while Scicode focuses on code snippets.Source: §2 · pp. 2–3
  • Many ML benchmarks focus on subcomponents or constrained settings, while RE-Bench uses only 7 hand-curated tasks and PaperBench [76] evaluates specific, well-defined research subcomponents.The cited benchmarks provide insights into specific ML tasks but generally do not capture realistic end-to-end workflows or offer a way to construct comprehensive tasks at scale.Source: §2 · p. 3

3 The EXP-Bench Benchmark and Dataset

EXP-Bench pairs paper-derived research tasks with ground-truth solutions and uses a semi-automated pipeline to construct and validate them. Its dataset spans diverse AI research areas, while its task structure and curation process support fine-grained evaluation.

3.1 EXP-Bench Dataset Specification

  • Each EXP-Bench task emulates a complete experimental process for a research question and includes an agent-facing problem statement and a ground-truth solution from the source paper and codebase.The task input includes a research question, high-level method, and relevant code repository; the ground truth specifies design, code modifications, and a conclusion based on results.Source: §3.1 · pp. 3–4
  • 461 tasks from 51 influential papers span Computer Vision, NLP, and Reinforcement Learning, with 53% sourced from NeurIPS 2024 and 47% from ICLR 2024.The dataset contains 12,737 individually gradable subtasks spanning design, implementation, and conclusion.Source: §3.1 · p. 4

3 The EXP-Bench Benchmark and Dataset

  • The pipeline addresses fragmented, domain-specific research artifacts with semi-automated task construction and lightweight human oversight.It links high-level ideas to executable implementations and is designed to support scalable construction of high-fidelity tasks.Source: §3 · p. 3, §3.2 · p. 4

3.2 EXP-Bench Semi-Automated Dataset Construction Pipeline

  • Source selection prioritizes influential papers from top-tier AI conferences that have public code repositories, using citation counts and repository activity as signals.The selected artifacts may still have imperfections, but represent significant and verifiable research contributions.Source: §3.2 · p. 4
  • Task extraction combines the research plan with its code implementation, decomposes tasks into explicit sub-steps, and uses multi-pass retrieval and subsection-level extraction to recover dispersed details.A tool-augmented agent searches the repository for required scripts and usage instructions, then traces validated scripts to produce natural-language implementation requirements.Source: §3.2 · pp. 4–5
  • Validation executes matched scripts in a clean container, checks traces against expected outputs, and returns failed tasks for refinement before lightweight human review.Tasks without matched implementations receive manual validation; finalized tasks include masked files to prevent shortcut access to original solutions.Source: §3.2 · pp. 5–6

4 Evaluation

Across EXP-Bench, agents often score below 30% on individual metrics, but strict end-to-end correctness is much rarer. Cost, stability, and failure analyses show additional weaknesses in experimental execution and reasoning.

4.1 Evaluating LLM-based Agents Performance on EXP-Bench: Setup & Main Results

  • The evaluation combines an integrity monitor (M), design (D), implementation (I), and conclusion (C) correctness scores with code-execution verification (E).The judge checks logs for disallowed behaviors, assesses outputs against ground truth, and validates generated code in an equivalent containerized environment.Source: §4.1 · p. 6
  • Scores stayed below 30% across metrics except RL implementation correctness (I), where some OpenHands models reached ≈41%; OH+o3-mini scored 1.4% on All✓.All✓ requires correctness in design, implementation, and conclusion; execution is an additional requirement for All·E✓.Source: §4.1 · pp. 6–7

4.2 Detailed Analysis

  • Runtime and cost correlated little with performance; OH+o3-mini offered the best trade-off, while OH+3.7 Sonnet was slowest and most expensive.IA models often used the full 40-minute allowance, whereas early stopping was common with OH models.Source: §4.2 · p. 7, Figure 6 · p. 8
  • Applying monitoring alone yielded 20.6% average accuracy, which fell to 3.7% with design and conclusion checks, 0.4% after implementation, and 0.2% after execution verification.These results use only tasks on which execution was run.Source: §4.2 · p. 7, Figure 6 · p. 8
  • Individual conclusion (C) and execution (E) scores show high variance, while conjunctive metrics such as C·D and I·E reduce variability.C·D filters conclusions not grounded in valid design plans; I·E discounts executions that do not fulfill setup requirements.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 Analysis on Prevalent Agent Failure Patterns

  • A two-pass analysis extracted domain-specific insights from error analyses, then grouped them into failure types, distilling 3,238 raw insights into 361 unique types.The paper presents a simplified subset of the resulting errors in Table 2.Source: §4.3 · pp. 7–8
  • Implementation failures led: 39.71% lacked essential components; execution failures included environment or dependency misconfigurations (29.38%) and script-level issues (23.84%).Examples include omitted retrieval or validation components, missing libraries, unregistered environments, unrecognized model names, and missing checkpoints.Source: §4.3 · p. 8
  • Design failures included incomplete or misclassified variables (16.05%); conclusion failures most often involved underdeveloped conclusions (26.18%) or incorrect interpretation (19.66%).Some agents added procedures or hyperparameters absent from the ground truth, or made performance claims without baseline comparisons.Source: §4.3 · p. 8

5 Discussion

EXP-Bench focuses on the experimentation procedure rather than the full AI research lifecycle. The authors identify broader lifecycle stages that remain outside its task structures and propose future work using dataset supervision.

5 Discussion

  • The benchmark covers experiment design through conclusions, but its task structures do not fully capture literature-gap identification, unstructured ideation, or real-world scientific discovery.The paper describes discovery as complex, iterative, and unpredictable.Source: §5 · p. 8
  • Table 2 gives a simplified subset of failure examples across experimentation phases, while Table 3 reports average benchmark scores for select task categories.Table 2 examples are measured across agent and model evaluations; Table 3's complete list is in Supplement D.Source: Table 2 · p. 9, Table 3 · p. 9
  • Future work will use EXP-Bench dataset supervision to improve automated experimentation, including reinforcement learning with verifiable rewards.The authors present this as a promising direction for enabling agents to navigate the research lifecycle and accelerate scientific discovery.Source: §5 · p. 9

6 Conclusion

EXP-Bench is presented as a benchmark and dataset for evaluating and guiding agents through end-to-end AI research experimentation. The related-work discussion distinguishes this focus from work on other stages of scientific discovery and from benchmarks centered on narrower or more deterministic tasks.

6 Conclusion

  • EXP-Bench evaluates end-to-end research experimentation and provides fine-grained tasks intended to guide agents through the workflow.Its tasks come from influential peer-reviewed papers and their codebases and cover design, implementation, execution, and conclusion derivation.Source: §6 · p. 9

A Extended Related Works

  • Automated scientific discovery has addressed hypothesis generation and code-assisted workflows, while rigor-sensitive experimentation remains underexplored.The discussion cites hypothesis-generation approaches [4], [84], and [103], and end-to-end workflows [50], [42], and [64].Source: A Extended Related Works · p. 16
  • Some automated discovery systems have limited evaluation or domain-specific validation, while EXP-Bench targets realistic, iterative research experiments.The discussion describes The AI Scientist [61] as focused on commonsense domains and the AI Co-scientist as mainly validated in biomedical areas.Source: A Extended Related Works · p. 16
  • Existing problem-solving benchmarks typically use well-defined tasks with deterministic solutions, unlike EXP-Bench’s iterative experimental challenges.The cited problem-solving benchmarks include, [26],, [77], and.Source: A Extended Related Works · p. 16

B Extended Details of the EXP-Bench Dataset

This appendix provides extended details of EXP-Bench, including its source-paper and subdomain listings and dataset access, and discusses societal effects, a category-score addendum, and examples analyzing agent performance.

B Extended Details of the EXP-Bench Dataset

C Societal Impact

  • Faster AI research may shorten innovation cycles and improve access and scientific efficiency, while increasing concerns about misuse, bias, governance, and researchers’ roles.The passage presents both potential benefits and risks of faster AI development.Source: C Societal Impact · p. 20

D Average Scores across All Paper Categories

  • The addendum provides updated IA+3.5 Haiku values for the Applications and Reinforcement Learning categories.Table 6 is described as giving average benchmark scores across selected task categories; its caption does not provide specific values.Source: D Average Scores across All Paper… · p. 21, Table 6 · p. 21

E Extended EXP-Bench Examples

  • The examples cover collaborative-perception detection under imperfect localization and TDNN implementation for speech recognition, detailing objectives, methods, source code, expected outcomes, and agent performance.The passage identifies these as two extended EXP-Bench examples.Source: E Extended EXP-Bench Examples · p. 24

E.1 Example 1: Robust Detection in Collaborative Perception

The examples illustrate EXP-Bench tasks in collaborative perception and speech recognition, each with an experimental objective, source code, and evaluation. Their agent outputs show different outcomes: one failed to complete the task after invalid operations, while the other built and integrated a TDNN pipeline.

E.1 Example 1: Robust Detection in Collaborative Perception

  • The collaborative-perception task tests whether HEAL maintains detection performance as Gaussian pose noise increases, holding OPV2V-H and the model architecture constant.Position and rotation noise levels are 0, 0.2, 0.4, and 0.6 meters/degrees; detection is measured with AP30, AP50, and AP70.Source: §E.1 · p. 24
  • The example agent’s core reconstructed file had a similar structure and functional goal to the source, but invalid operations in another file prevented task completion.Figure 9 shows the Git diff between the masked source file and reconstructed code; Figure 10 presents the failure analysis.Source: §E.1 · p. 24, Figure 9 · p. 27, Figure 10 · p. 28

E.2 Example 2: Time Delay Neural Network for ASR

  • The ASR task specifies a three-Conv1d-layer TDNN and targets test-set WER below 1%.The setup holds the yesno dataset, CTC loss, and 23-dimensional fbank features constant, and uses k2 one_best_decoding.Source: §E.2 · p. 29
  • The example agent correctly built the TDNN and integrated it with training and decoding, though its model differed from the reference in layer configuration and parameter initialization.Figure 13 shows the diff between the reconstructed model.py and the original implementation.Source: §E.2 · p. 29, Figure 13 · p. 32

F Dataset Curation and Benchmark Construction Details

The curation pipeline was iteratively tested and refined with collaborator inspection, and the final pipeline reduced manual validation effort. The paper also reports per-paper extraction costs and documents extraction issues addressed during pipeline development.

F Dataset Curation and Benchmark Construction Details

  • The pipeline was iteratively prototyped on real papers and refined through collaborator inspection to address curation failure modes.Manual validation used the papers and open-source repositories as ground truth, and representative patched issues are listed in Table 7.Source: F Dataset Curation and Benchmark… · p. 33, Table 7 · p. 33
  • After refinement, manual validation fell from roughly two hours to around 20 minutes per paper.The remaining checks primarily confirmed alignment with source materials, with only rare minor adjustments.Source: F Dataset Curation and Benchmark… · pp. 33–34
  • Extraction costs averaged approximately $60 USD per paper, with costs driven primarily by input tokens for full papers and codebases.The pipeline used o3-mini-2025-01-01-preview for main task extraction and claude-3-7-sonnet-20250219-v1:0 for implementation extraction.Source: F Dataset Curation and Benchmark… · p. 34

G Extended Analysis on Prevalent Agent Failure Patterns

The failure analysis catalogs omissions and errors across experimental design, execution, and implementation, with setup failures spanning many components. The category classification may overlap because it was performed by an LLM.

G Extended Analysis on Prevalent Agent Failure Patterns

  • Some categories may overlap because an LLM performed the classification.Source: G Extended Analysis on Prevalent Agent… · p. 35
  • Design failures include incomplete experiment specifications and missing or partial design-variable and hyperparameter details.The listed categories also include missing formatting, variable labels, and configuration-file variables.Source: G Extended Analysis on Prevalent Agent… · p. 35
  • Execution failures include environment and dependency problems, script or file errors, and missing dependencies or setup scripts.The listed categories also include tensor-operation, syntax, indexing, and data-file errors.Source: G Extended Analysis on Prevalent Agent… · p. 36
  • Setup failures range from missing essential implementation components to incomplete metrics, data preparation, training, evaluation, and reproducibility.The listed categories also cover missing model architectures, configuration loading, validation, and multiple runs for statistical significance.Source: G Extended Analysis on Prevalent Agent… · pp. 36–37
  • The setup taxonomy also lists many specific omissions and incorrect implementations, including visualization, data handling, model integration, and checkpointing.Further examples include missing evaluation routines, logging, configuration, and training components.Source: G Extended Analysis on Prevalent Agent… · pp. 38–40

H Evaluation Judge

The evaluation framework uses an LLM-based judge to assess integrity, experimental design, implementation, and conclusions. Its prompts specify checks against logs and task ground truth, while long inputs are evaluated in chunks.

Evaluation Judge

  • An LLM-based judge using o3-mini-2025-01-01-preview assesses integrity, experimental design, implementation correctness, and conclusion validity.The paper provides the judge’s full system prompts.Source: H Evaluation Judge · p. 41
  • Long inputs such as Git diffs and agent logs are split into chunks, with results and relevant context carried forward between chunks.Source: H Evaluation Judge · p. 41
  • The monitor checks logs for paper access, Git operations, and fake, hardcoded, mock, or reused data in place of experimental results.Its prompt requests evidence and reasoning for the decisions.Source: H Evaluation Judge · pp. 41–42
  • Design is scored by matching predicted variables against ground truth, while conclusions are judged for semantic match to the ground-truth conclusion.The prompt returns a percentage score for design and a correct-or-incorrect conclusion judgment.Source: H Evaluation Judge · p. 42
  • Implementation is evaluated by comparing the agent’s Git diff with setup requirements, using reference scripts to judge whether requirements are fulfilled.The instructions prioritize intent over exact filename or function-name matches and score the fulfilled requirements.Source: H Evaluation Judge · p. 43
  • Figure 14 indicates that stricter metrics reveal lower true correctness.Source: Figure 14 · p. 45

I Additional Analysis

The section points readers to §4.2 for detailed analysis breakdowns.

I Additional Analysis

I.1 Conjunctive Evaluation Metrics Analysis

The additional analysis gives the full evaluated-agent set for the conjunctive-metrics analysis and summarizes cost–time statistics, with a caveat about interpreting low values.

I.1 Conjunctive Evaluation Metrics Analysis

  • Figure 14 includes all evaluated agents and models, unlike the subset shown in Figure 6b.Source: §I.1 · p. 45

I.2 Cost-Time Distribution

  • Some runs exceeded the 40-minute soft timeout, and failed experiments can produce unusually low time and cost values.Source: §I.2 · p. 45
Loading 2505.24785v2…