PaperlayerView exact revision on arXiv

Source-linked AI summary

From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking

Gyeongwon James Kim, Alex Wilf, Louis-Philippe Morency, Daniel Fried

arXiv:2506.19724v1cs.AI

TL;DR

Existing benchmarks do not evaluate scientific experiment implementation across varied amounts of available code. AutoExperiment tests agents on masked-code experiments, finding performance degrades as masking increases and that multi-trial success exceeds single-trial success by over 10%.

  • Problem

    Existing benchmarks do not evaluate agents implementing scientific experiments from papers with varied portions of the original codebase.

    Source: §1 · p. 1

  • Method

    AutoExperiment gives agents a paper, run command, and codebase with functions masked, then evaluates their implementation by running the experiment and comparing results.

    Source: §1 · p. 1

  • Results

    Performance degrades sharply as more functions are masked, while pass rate improves over 10% from Pass@1 to Pass@5.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The Pass@1-to-Pass@5 gap motivates verifier approaches to selecting among agent attempts.

    Source: §1 · p. 2

    The benchmark covers machine-learning papers and Python code, so generalizability to other scientific domains and programming languages is unclear.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent progress in autonomous code generation has fueled excitement around AI agents capable of accelerating scientific discovery by running experiments. However, there is currently no benchmark that evaluates whether such agents can implement scientific ideas when given varied amounts of code as a starting point, interpolating between reproduction (running code) and from-scratch replication (fully re-implementing and running code). We introduce AutoExperiment, a benchmark that evaluates AI agents' ability to implement and run machine learning experiments based on natural language descriptions in research papers. In each task, agents are given a research paper, a codebase with key functions masked out, and a command to run the experiment. The goal is to generate the missing code, execute the experiment in a sandboxed environment, and reproduce the results. AutoExperiment scales in difficulty by varying the number of missing functions $n$, ranging from partial reproduction to full replication. We evaluate state-of-the-art agents and find that performance degrades rapidly as $n$ increases. Agents that can dynamically interact with the environment (e.g. to debug their code) can outperform agents in fixed "agentless" harnesses, and there exists a significant gap between single-shot and multi-trial success rates (Pass@1 vs. Pass@5), motivating verifier approaches to our benchmark. Our findings highlight critical challenges in long-horizon code generation, context retrieval, and autonomous experiment execution, establishing AutoExperiment as a new benchmark for evaluating progress in AI-driven scientific experimentation. Our data and code are open-sourced at https://github.com/j1mk1m/AutoExperiment .

1 Introduction

Existing benchmarks do not test agents implementing scientific experiments from papers with varied amounts of original code. AutoExperiment progressively masks functions and finds that performance falls as difficulty increases, while dynamic interaction and multiple trials can improve outcomes.

1 Introduction

  • Prior benchmarks either assess related code-generation tasks, reproduction without implementation, or replication from paper descriptions, and use fixed difficulty.They do not evaluate implementing experiments from a paper with varied portions of the original codebase.Source: §1 · p. 1
  • AutoExperiment evaluates agents by having them implement masked functions, run experiments, and report results against the published code’s outcomes.Agents receive the paper, a run command, and a codebase with a progressively increasing number of functions masked.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Agent performance falls quickly as more code is masked, while dynamic interaction and Pass@5 improve performance over fixed interaction and Pass@1, respectively.The Pass@1-to-Pass@5 pass-rate gap exceeds 10%.Source: §1 · p. 2

2 The AUTOEXPERIMENT Benchmark

AutoExperiment builds tasks from reproducible, previously replicated machine-learning papers, masking core Python functions and testing whether agents can restore experimental results. It varies task difficulty through masking and gives agents commands and an opportunity to debug before submission.

2 The AUTOEXPERIMENT Benchmark

  • The benchmark selects four peer-reviewed papers previously replicated in the ML Reproducibility Challenge and checks that their code reproduces reported results.Repositories are filtered for maintenance, documentation, and a strong mapping between paper content and code.Source: §2 · pp. 2–3
  • Masking n functions creates unique samples from 85 core functions, with up to 275,990 possible samples at n = 5.Each masked function is replaced with NotImplementedError; evaluation uses at most 100 samples per n.Source: §2 · p. 3
  • The harness gives agents repository commands that depend on masked functions, lets them debug, and checks reported results against gold-standard runs.A sample passes when all test results are within 5% of gold-standard values; paper numerical results are removed to prevent cheating.Source: §2 · p. 3
  • Table 1 organizes possible samples and average test cases per sample by n, while evaluation is capped at 100 samples per setting.The average number of test cases increases monotonically with n.Source: Table 1 · p. 3, §2 · p. 3
  • When training experiments would run too long, some test cases shorten training while checking that the curtailed setting discriminates passing from failing solutions.A complete GPT-4o run averaged just over 11 minutes per sample, including agent actions and experiment runtime.Source: §2 · p. 3
  • The agent setup varies tools, prompting strategies, history management, and backbone LLMs across experiments.Main experiments use ReAct prompting with full history management; tested backbones include GPT-4o, GPT-4o-mini, Claude-3.5-Sonnet, and Claude-3.7-Sonnet.Source: §2 · pp. 3–4
  • Experiments run in Docker with preconfigured conda environments, and agents are limited to 50 action steps, 30 minutes, and $1 of LLM compute per sample.The setup uses a Linux machine with an AMD 24-Core processor and two NVIDIA GeForce RTX 3090 Ti GPUs.Source: §2 · p. 4

3 Experiments

Across the experiments, success falls sharply as more functions are masked, while dynamic interaction and paper context become increasingly useful. Multiple attempts also reveal substantial Pass@1-to-Pass@5 gains, although model-based verification captures only part of that potential.

3.1 From Reproduction to Replication: Removing n ≥1 Functions

  • At n = 2, pass rates fall 70–90% on average from n = 1, and most models reach negligible pass rates by n = 5.Claude-3.7-sonnet and Claude-3.5-sonnet each achieve 9.6% at n = 2, compared with 35.3% for GPT-4o at n = 1.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 The Pass@k Gap

  • Pass@1-to-Pass@5 rises from 35.3% to 48.2% for GPT-4o and from 31.8% to 42.2% for Claude 3.5 Sonnet.Model-based verifiers improve performance somewhat but remain substantially below oracle-level verification.Source: Figure 3 · p. 5, §3.2 · p. 5

3.3 Scaling Interactivity and Test-Time Compute in Agents

  • Dynamic interaction raises GPT-4o pass rate from 8.3% to 35.3% and o3-mini from 27.8% to 33.3%, outperforming fixed interaction for both.The authors suggest debugging may partially explain the dynamic agents’ advantage: 29.1% of runs recover from crashes, and 18.6% end with correct working code.Source: Table 3 · p. 6, §3.3 · p. 6
  • Scaling reasoning tokens in the fixed setting yields moderate gains, while scaling dynamic interactions performs better and saturates around 38.9%.For fixed interaction, o1 and o3-mini peak at 22.2% and 27.8%, respectively.Source: Figure 4 · p. 6, §3.3 · p. 7

3.4 Dependence on Natural Language

  • At n = 1, Full context achieves 35.3% and No context 34.1%; without paper context, performance drops to complete failure by n = 4.The authors report that paper context becomes more important as more functions are masked, while relevant code context also matters.Source: Figure 5 · p. 7, §3.4 · pp. 7–8

4 Related Work

Prior work explores generative agents and code-generation benchmarks, but existing research-agent benchmarks do not test implementing research ideas by masking varying amounts of gold-standard code.

4 Related Work

  • As agents gain capabilities for idea generation [Qi et al., 2023, Wang et al., 2024a], peer review [Tyser et al., 2024, Liu and Shah, 2023], and writing assistance [Altmäe et al., 2023, Salvagno et al., 2023], benchmarking their ability to run experiments becomes more important.The paper places this motivation alongside work on AI for scientific research [Simon, 1977, Langley, 2000] and agents running experiments [Lu et al., 2024].Source: §4 · p. 8
  • Generative agents face inherited challenges in retrieval, reasoning, planning, and reflection, as well as agent-specific challenges in tool use and managing long interaction histories.The paper situates these challenges within work on interactive agents [Xi et al., 2023], inherited backbone challenges [Shinn et al., 2023, Madaan et al., 2023], and agent-specific challenges [Wang et al., 2024c, Zhang et al., 2024].Source: §4 · p. 8
  • Existing code-generation benchmarks cover docstrings, notebooks, and GitHub issue patches, while related research-agent benchmarks target model improvement, research tasks, repository reproduction, or full replication from scratch.None tests implementing research ideas with varying amounts of gold-standard code masked out.Source: §4 · p. 8

5 Conclusion

The conclusion presents AUTOEXPERIMENT as a benchmark spanning reproduction to full replication, and reports sharp performance degradation with increased autonomy alongside benefits from dynamic decision-making and multiple trials.

5 Conclusion

  • AUTOEXPERIMENT evaluates agents implementing and running scientific experiments from paper descriptions, with increasing function removal bridging reproduction and full replication.The conclusion also reports that dynamic decision-making is critical and multi-trial accuracy substantially exceeds single-trial accuracy, motivating verifiers.Source: §5 · p. 8

Limitations

The benchmark is limited to machine-learning papers and Python code, with broader generalizability unclear; the appendices describe its paper collection and a proxy evaluation using curtailed test cases.

Limitations

  • The benchmark covers machine-learning papers and Python code; its generalizability to other scientific domains and programming languages is unclear.The dataset has 85 tasks at n=1 and is resource-intensive to run, though parallelization is possible.Source: Limitations · p. 9

A Papers

  • Table 4 lists the papers included in AUTOEXPERIMENT.The supplied passage gives the table's scope but not its entries.Source: Table 4 · p. 13

B Curtailed Test Cases

  • Curtailed test cases had very high precision and recall against full test cases in a 92-run comparison, while taking significantly less time and compute.The authors report that this makes curtailed tests an effective proxy and supports large-scale testing.Source: B Curtailed Test Cases · p. 13, Table 5 · p. 13

C Agent Architectures & Backbone Models

The section compares prompting and interaction-history strategies, finding that their performance depends on the model and context capacity; ReAct with GPT-4o and full memory performs best among the reported configurations.

C Agent Architectures & Backbone Models

  • ReAct with GPT-4o and full memory achieves a 38.9% success rate, while the best-versus-worst comparison is significant (p-value 0.004) and best-versus-second-best is not (p-value 0.243).GPT-4o-mini benefits from summarization rather than full context with ReAct, while Planning prompts benefit from summarization strategies, especially for smaller models.Source: Table 6 · p. 13, C Agent Architectures & Backbone Models · p. 14
  • The evaluated prompting approaches are ReAct, which elicits a thought and next action, MLAgentBench's structured plan-and-reflection response [Huang et al., 2024], and Planning-Only's step-by-step plan and status [Song et al., 2023].The paper examines these approaches alongside alternative interaction-history management strategies.Source: C Agent Architectures & Backbone Models · p. 13
  • History management ranges from passing the full interaction record to a sliding window of recent interactions or recent interactions plus a summary of older ones.The latter two approaches are denoted Sl(k) and Sm(k), respectively.Source: C Agent Architectures & Backbone Models · pp. 13–14

D Evaluation Samples for Higher Values of n

The evaluation samples for higher masking levels are selected to manage the exponentially growing number of possible samples. The verifier prompt asks an LLM to compare function implementations and select the best candidate.

D Evaluation Samples for Higher Values of n

E Verifier Prompt

  • The verifier prompt asks an expert software engineer to choose the best LLM-generated implementation of a function.It also requests an explanation of the choice.Source: E Verifier Prompt · p. 14

F Code Context Retrieval Results

Embedding-based retrieval achieves the highest reported pass rate among the compared retrieval strategies. The results also identify code retrieval as a central challenge for the benchmark.

F Code Context Retrieval Results

  • 41.7% pass rate with embedding-based retrieval exceeds 36.1% with in-context retrieval and 33.3% with AST-based retrieval.The comparison is reported in Table 8.Source: F Code Context Retrieval Results · p. 15
  • Placing all relevant code in the context window degrades performance relative to the reported retrieval strategies.The passage describes this as underscoring limitations of context-window-based reasoning.Source: F Code Context Retrieval Results · p. 15
  • The results identify code retrieval as a central challenge in AutoExperiment and an area for future research.The authors connect this conclusion to the reported retrieval-strategy comparison.Source: F Code Context Retrieval Results · p. 15

G Agent-less Experiment

The agentless harness retrieves relevant paper and repository context, prompts an LLM to fill a masked function, and validates the generated code using experiment test cases. Its prompts separately support file localization, reasoning, and code completion.

G Agent-less Experiment

  • The agentless harness retrieves natural-language and code context, fills the masked function, then validates the code and extracts experiment results.Natural-language retrieval uses the function header and docstring to find paper snippets; code retrieval uses them to find repository snippets. Code infilling follows Muennighoff et al. [2025].Source: G Agent-less Experiment · p. 15

G.1 Code retrieval file localization prompt

  • The file-localization prompt requests the top 10 repository paths related to the code, given the repository structure and function header and docstring.The requested output is a newline-separated list of paths.Source: §G.1 · p. 15

Reasoning prompt

  • The reasoning prompt supplies paper context, code context, and file content before asking the model to consider how to implement the missing Python function.The prompt frames the task as implementing one missing function.Source: Reasoning prompt · p. 16

Completion prompt

  • The completion prompt provides the model’s thought and file content, then requests only the missing function implementation.The prompt specifies a Python function as the output.Source: Completion prompt · p. 16

H Test Time Compute Experiments

The test-time-compute experiment varies the reasoning-token budget for an agentless harness with o1 and o3-mini. The paper also describes interactive-agent prompts and tools for modifying code and running experiments.

H Test Time Compute Experiments

  • The fixed agentless setting varies maximum reasoning tokens from 0 to 16384 for o1 and o3-mini, prompting for reasoning before code infilling.If reasoning ends before the target token count, the harness appends a “Wait” token; Table 9 presents scaling in the fixed setting.Source: H Test Time Compute Experiments · p. 16, Table 9 · p. 16

I.1 Initial Prompt

  • The initial prompt asks the research assistant to implement missing functions and run bash refsol.sh to obtain experiment results.The environment is described as already set up, with some functions not implemented.Source: §I.1 · p. 17

ReAct

  • The ReAct prompt tells the agent to consider what action to perform next.The supplied prompt fragment gives no further procedure.Source: ReAct · p. 17

MLAgentBench

  • The MLAgentBench prompt asks for reflection on observations and errors, plus a research plan and fact-check of reported progress.It says performance numbers can be confirmed only by running code and observing the output.Source: MLAgentBench · p. 17

Planning-Only

  • The planning-only prompt requests a high-level plan with current status and confirmed results.The prompt excerpt provides no further planning requirements.Source: Planning-Only · p. 17

I.3 Tool Calls

  • The agent’s standard tool set supports repository navigation, file reading and editing, and Python or bash execution.The tools are described as inspired by MLAgentBench [Huang et al., 2024] and SWE-Agent [Yang et al., 2024].Source: §I.3 · p. 17

J Compute Resources

A GPT-4o ReAct/full-memory trajectory traces masked-code repair through execution and result reporting, using paper context to implement an RRR bounding-box loss and correct a device mismatch.

K Example Trajectory

Step 2

  • RRR* extends RRR to arbitrary attribution methods and regularizes attributions outside ground-truth masks, without explicitly encouraging high importance inside them.Unlike RRR, RRR* does not introduce a uniformity prior.Source: Step 2 · p. 23

Step 3

Step 4

  • The implementation builds a bounding-box mask, normalizes attributions, and sums their squared values outside the boxes.The trajectory uses the mask complement, 1−M, to select the outside-box attributions.Source: Step 4 · pp. 24–25

Steps 5–6

  • The first run failed because tensors were split across cuda:0 and CPU; the agent moved the mask to the attribution tensor's device before retrying.The reported runtime error was a device mismatch during the loss calculation.Source: Step 5 · p. 25, Step 6 · pp. 25–26

Step 7

  • The example's first experiment used 10 training examples, 100 validation examples, and 100 test examples, and reported BB-IoU among its validation metrics.The listed validation metrics also include accuracy and average loss.Source: Step 7 · p. 26

Step 8

  • BB-IoU was 0.0009418611755977625 for Experiment 1 (BCos), 0.09110355804335622 for Experiment 2 (GradCam), and 0.0013012630506170516 for Experiment 3 (IxG).The trajectory identifies these as the three experiments' IoU results.Source: Initial Prompt · p. 22, Step 8 · p. 26
Loading 2506.19724v1…