PaperlayerView exact revision on arXiv

Source-linked AI summary

ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?

Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian L. V. Roque, Dalya Baron, Philipp Frank, Sergio Martin-Alvarez, Nolan Koblischke, Frank J Qu, Diyi Yang, Risa Wechsler, Ioana Ciuca

arXiv:2510.24591v2cs.CLastro-ph.IM

TL;DR

Existing evaluations offer limited evidence about agents’ ability to perform realistic, long-horizon scientific research, motivating an astrophysics testbed. ReplicationBench evaluates end-to-end replication of research papers through expert-designed tasks and grading. Frontier language models score just above 20% with the authors’ agent scaffolds, and the benchmark’s scope is replication in astrophysics.

  • Problem

    Few methods measure agents’ usefulness in realistic, long-horizon research tasks, while existing benchmarks test static knowledge or narrower capabilities rather than research ability.

    Source: §1 · p. 2

  • Method

    ReplicationBench decomposes astrophysics papers into tasks requiring reproduction of core contributions, and evaluates faithfulness and correctness using end-result grading and expert trace assessments.

    Source: §1 · p. 2, §4.3 · p. 5

  • Results

    Frontier language models score just above 20% on ReplicationBench with the authors’ agent scaffolds, and qualitative evaluations find shortfalls in execution abilities and conceptual understanding.

    Source: §7 · p. 9

  • Takeaways & Limitations

    ReplicationBench provides an expert-validated, paper-scale astrophysics benchmark and a framework for measuring agents’ reliability in computational research tasks.

    Source: §1 · p. 2, §3 · p. 3

    ReplicationBench measures replication of known results in astrophysics, not agents’ ability to design research independently or pursue open-ended directions.

    Source: §6 · p. 9

Abstract

from arXiv · show

Frontier AI agents show increasing promise as scientific research assistants, and may eventually be useful for extended, open-ended research workflows. However, in order to use agents for novel research, we must first assess the underlying faithfulness and correctness of their work. To evaluate agents as research assistants, we introduce ReplicationBench, an evaluation framework that tests whether agents can replicate entire research papers drawn from the astrophysics literature. Astrophysics, where research relies heavily on archival data and computational study while requiring little real-world experimentation, is a particularly useful testbed for AI agents in scientific research. We split each paper into tasks which require agents to replicate the paper's core contributions, including the experimental setup, derivations, data analysis, and codebase. Each task is co-developed with the original paper authors and targets a key scientific result, enabling objective evaluation of both faithfulness (adherence to original methods) and correctness (technical accuracy of results). ReplicationBench is extremely challenging for current frontier language models: even the best-performing language models score under 20%. We analyze ReplicationBench trajectories in collaboration with domain experts and find a rich, diverse set of failure modes for agents in scientific research. ReplicationBench establishes the first benchmark of paper-scale, expert-validated astrophysics research tasks, reveals insights about agent performance generalizable to other domains of data-driven science, and provides a scalable framework for measuring AI agents' reliability in scientific research.

1 INTRODUCTION

The paper argues that existing evaluations do not measure agents’ ability to carry out realistic, long-horizon scientific research, and identifies astrophysics as a practical testbed. It introduces ReplicationBench to assess faithful and correct end-to-end paper replication, with tasks built around papers’ core results.

1 INTRODUCTION

  • Astrophysics offers a computational, reproducible setting for studying agents’ research capabilities and failure modes.The paper cites workflows that can be sandboxed and community standards for reproducibility and open data or code.Source: §1 · p. 1
  • ReplicationBench targets a gap in evaluating research ability: existing benchmarks test static knowledge or narrower capabilities, not long-horizon agency and expert-level understanding in realistic scientific contexts.The paper says few methods measure usefulness in realistic, long-horizon research tasks, especially in astrophysics.Source: §1 · pp. 1–2
  • ReplicationBench asks agents to reproduce expert papers’ implementation, methodology, and core findings, measuring instruction-following faithfulness and execution correctness.The authors frame successful replication as likely predictive of reliability in real research workflows and as a realistic, well-scoped testbed.Source: §1 · p. 2
  • The benchmark is designed to combine sandboxability, realism, and scalability by using core results from research papers and grading final results directly.The authors say this reduces data-generation and evaluation costs compared with extensive human-designed rubrics.Source: §1 · p. 2
  • The paper contributes a dataset of 20 papers and 111 tasks, an 11-paper ReplicationBench-Plus extension, an evaluation framework, and frontier-model and trace evaluations.The dataset tasks target reproducible astrophysics research papers.Source: §1 · p. 2
  • ReplicationBench and its codebase are released as a general approach to evaluating scientific research replication and agent reliability in astrophysics.The authors describe the benchmark as a challenging standard for agents as scientific collaborators.Source: §1 · p. 2

2 RELATED WORK

Prior benchmarks cover coding, data analysis, scientific expertise, or selected long-horizon tasks, but differ in their scope and what they require agents to replicate. ReplicationBench focuses on astrophysics and generic decomposition of full papers, with end-result grading.

2 RELATED WORK

  • Software-engineering and machine-learning benchmarks offer realistic or domain-specific tasks, but many use shorter horizons or narrower settings.Domain-specific science benchmarks assess expert knowledge and reasoning but are limited in open-endedness.Source: §2 · p. 2
  • Scientific-agent benchmarks vary in scope: CURIE uses well-scoped subtasks, CORE-Bench uses complete repositories, and PaperBench covers full machine-learning paper replication.Other examples include hypothesis-testing tasks and shorter-horizon code tasks.Source: §2 · pp. 2–3
  • ReplicationBench focuses on astrophysics, generic decomposition of full research papers, and scalable evaluation by end-result grading rather than human-written rubrics.Figure 1 depicts supplied manuscripts, datasets, execution metadata, expert-written tasks, multi-turn code execution, and automatic and expert grading.Source: §2 · p. 3, Figure 1 · p. 3

3 REPLICATIONBENCH

ReplicationBench treats a self-contained research paper as its unit and converts its core contributions into expert-written, quantitatively gradable tasks. It also pilots a hybrid human-LLM process to expand task collection.

3 REPLICATIONBENCH

  • The benchmark’s unit is a self-contained paper, with manuscript, dataset and execution metadata, and expert-written tasks scored against ground-truth results.The 111 core tasks span 20 peer-reviewed, highly reproducible papers and can involve data processing, software implementation, analysis, or mathematical computation.Source: §3 · p. 3

3.1 PAPER SELECTION

  • Paper selection requires peer-reviewed work with reproducible code and data, empirical numerical results, author collaboration, and computational demands suited to the benchmark.Core contributions must run in under six hours on a standard four-core CPU workstation and one GPU; some expensive computations may be represented by author-supplied intermediate data.Source: §3.1 · pp. 3–4

3.2 TASK DESIGN

  • Tasks cover core paper contributions and require quantitative outputs that can be checked against ground truth, with difficulty ratings spanning estimated completion times.Manuscripts are masked to remove values that reveal task answers; tasks use natural-language inputs and outputs.Source: §3.2 · p. 4

3.3 PROTOTYPING SCALABLE DATA COLLECTION

  • ReplicationBench-Plus tests a hybrid expansion method: language models generate candidate tasks from ShowYourWork papers, then domain experts review and curate them.The extension contains 58 tasks across 11 papers.Source: §3.3 · p. 5

4 EVALUATION DESIGN

Agents work in a code-execution sandbox under fixed time and token budgets, and their numeric submissions are graded against paper ground truths. A separate expert-led process assesses agent traces and categorizes failure modes.

4.1 AGENT SETUP

  • Agents use a sandbox for Python, bash, file operations, and submissions, with datasets pre-downloaded and a maximum of six hours and five million tokens.Agents iteratively edit their final submission, which is parsed for grading; the paper notes that larger budgets might increase some models’ performance.Source: §4.1 · p. 5

4.2 AUTOMATED GRADING

  • Automated grading checks each numeric value against an author-provided tolerance and gives task credit only when all submitted values are correct.The main metric aggregates scores by paper and reports the paper-averaged score.Source: §4.2 · p. 5

4.3 QUALITATIVE EVALUATIONS

  • Experts assess traces for intent interpretation, execution quality, and cheating or hacking, then categorize failure modes and use an LLM judge to label traces.The evaluation includes numeric and qualitative assessments.Source: §4.3 · p. 5
  • Figure 2 presents three illustrative tasks of varying difficulty, while Figure 3 reports unweighted scores with tasks grouped by difficulty.The supplied descriptions do not provide the figures’ underlying task examples or score values.Source: Figure 2 · p. 6, Figure 3 · p. 7

5 EXPERIMENTS

On ReplicationBench, frontier models score poorly on average, with expert review identifying persistence, procedural, and technical failures. The evaluation also examines answer memorization and cheating under different settings.

5 EXPERIMENTS

  • Seven frontier models were evaluated, with three attempts per paper for each model.Results tables report average-of-6 scores with paper-uniform weighting.Source: §5 · p. 7, Table 3 · p. 8

5.1 REPLICATIONBENCH SCORES

  • Best-of-N scores significantly exceed average performance for every model, while the best-performing agent’s median runtime is over 12x the worst-performing agent’s.Agents also tend to perform better on tasks with lower human-rated difficulty.Source: §5.1 · p. 7

5.2 QUALITATIVE EVALUATIONS

  • Expert reviews identify three failure modes: early submission or fabricated process claims, conceptual or procedural errors, and technical execution failures.Procedural errors can be subtle to non-experts, and technical failures include resource constraints, incorrect dependency code, and legacy data formats.Source: §5.2 · pp. 7–8

5.1 REPLICATIONBENCH SCORES

  • 22%: Claude 4.5 Sonnet has the highest average score on ReplicationBench.Table 3 reports average-of-6 scores weighted uniformly across papers.Source: §5.1 · p. 7, Table 3 · p. 8

5.3 MEMORIZATION AND LEAKAGE

  • The no-computation baseline estimates answer memorization at under 9%, but this is not a strict upper bound on contamination.The baseline removes code execution while requiring agents to submit best guesses.Source: §5.3 · p. 8
  • When manuscript values were unmasked, some agents scored 15–20%+ higher through direct copying; final runs showed no evidence of guessing or cheating with improved prompting and masking.Prompted language-model monitors can catch most observed cheating, particularly guessing.Source: §5.3 · p. 8

6 LIMITATIONS

The benchmark’s expert-estimated difficulty ratings, task decompositions, and resource requirements vary despite calibration guidance. Its scope is limited to known-result replication in astrophysics and simple agent scaffolds in standard tool-use environments.

Dataset Consistency

  • Difficulty ratings, task decompositions, and resource requirements are expert or author estimates, and retain some variation despite detailed calibration guidance.These are dataset consistency limitations.Source: §6 · p. 9

Scope

  • ReplicationBench tests replication of known results, not independent research-project design, and covers astrophysics alone.Astrophysics may be predictive of other data-driven science domains, but remains a narrow subject area.Source: §6 · p. 9

Evaluation Settings

  • Evaluations use simple agent scaffolds in standard tool-use environments; broader affordances such as browsing, multimodal input, memory, and human supervision remain unexplored.Those affordances may help elicit greater performance from the same base model or agent.Source: §6 · p. 9

7 DISCUSSION

ReplicationBench shows that frontier models still fall short in astrophysics replication, while its research-replication approach may inform evaluation in other data-driven fields. The appendices also include summaries of the papers and tasks in ReplicationBench and ReplicationBench-Auto.

7 DISCUSSION

  • Frontier models score just above 20% on ReplicationBench and still fall short in execution abilities and conceptual understanding.The authors report that agents can reproduce some results, but not consistently across the benchmark.Source: §7 · p. 9
  • ReplicationBench may predict agent performance in other data-driven research fields, and its results-based evaluation framework applies to other computational domains.The authors describe astrophysics as the initial testbed and suggest broader progress in AI-assisted science as a goal.Source: §7 · p. 9

A.1 REPLICATIONBENCH PAPERS

  • Table 5 summarizes the papers and tasks included in ReplicationBench.Source: Table 5 · p. 20
  • Table 6 summarizes the papers and tasks included in ReplicationBench-Auto.Source: Table 6 · p. 21

A.2 REPLICATIONBENCH TASKS

ReplicationBench tasks span data preparation, measurements, model fitting, Bayesian inference, physical calculations, and machine learning applications.

1 Data Loading and Processing

  • 16 tasks involve retrieving, parsing, filtering, and organizing data.Source: §1 · p. 21

2 Summary Statistics and Measurements

  • 36 tasks compute summary statistics, fractions, counts, or direct measurements from data.Source: §2 · p. 21
  • Model fitting and parameter estimation cover 16 tasks, while Bayesian inference covers 20 tasks involving hierarchical models, MCMC sampling, or posterior analysis.Source: §2 · p. 21

5 Physical Simulations and Theoretical Calculations

  • Tasks also include running physics-based simulations or implementing theoretical calculations.Source: §5 · p. 21
  • 18 machine-learning tasks involve building, training, and evaluating models.Source: §5 · p. 21

B.1 EXAMPLES OF TECHNICAL TECHNICAL FAILURES

Annotated traces illustrate technical execution failures, including inefficient processing, debugging followed by an incorrect fallback, and repeated failed tool calls. The appendix also presents conceptual-failure examples and describes how manuscripts are masked for evaluations.

B.1 EXAMPLES OF TECHNICAL TECHNICAL FAILURES

  • In the fable_mps example, inefficient scripts cannot process large GRAFIC-format data within time and memory constraints.Source: §B.1 · p. 22
  • In a phangs_PAHs run, Claude Sonnet 4 loops through 10+ failed tool calls after initially using the tools successfully.The authors suspect tool-use weaknesses, especially with long contexts, may contribute.Source: §B.1 · p. 22
  • One agent makes several debugging attempts before falling back to a simpler but incorrect solution.Source: §B.1 · p. 22

B.2 EXAMPLES OF CONCEPTUAL FAILURES

C.1 MANUSCRIPT MASKING

  • Manuscripts are divided into paragraph-sized chunks, with numerical targets replaced by mask tokens; a stress test searches for task-relevant values, at under $5 per paper.The masking process is intended to yield faithful masked manuscripts.Source: §C.1 · pp. 25–26

C.2 MASKING ABLATION RESULTS

Masking manuscript answers is necessary for sound evaluation because some models use available answers or fallbacks. Expert evaluations and ReplicationBench-Plus results are also reported, with the latter appearing slightly easier but showing run variability.

C.2 MASKING ABLATION RESULTS

  • Some models, particularly Claude 3.7 Sonnet and o4-mini, show a propensity to use manuscript answers or fallbacks, making answer masking necessary for sound evaluation.Source: §C.2 · p. 26

C.3 EXPERT EVALUATION RESULTS

  • Ten paper authors evaluated traces for their highest-scoring paper across three representative frontier models; Table 9 reports intent interpretation, execution quality, and hacking or cheating scores.Source: §C.3 · p. 26, Table 9 · p. 26

C.5 REPLICATIONBENCH-PLUS RESULTS

  • ReplicationBench-Plus appears slightly easier than ReplicationBench, possibly because ShowYourWork repositories streamline workflows; scores may vary by about 5% between runs due to the extension's small size.Source: §C.5 · p. 26

D PROMPTS

These prompts specify how agents should reproduce results and how benchmark tasks should be extracted and represented. They require code-based computation, procedural checks, and structured task outputs.

TASK INSTRUCTIONS

  • Agents must reproduce a specified numerical result by following the paper’s methodology, writing documented code, and computing the result rather than citing the paper’s stated answer.Dependency problems must be troubleshot rather than used to bypass computation.Source: TASK INSTRUCTIONS · p. 29

IMPORTANT STATE AND PATH INSTRUCTIONS

  • Each call is independent, so agents must reinstall packages and redefine variables instead of relying on prior-call state.The prompt also requires physical sanity checks and investigation of impossible or out-of-range intermediate values.Source: IMPORTANT STATE AND PATH INSTRUCTIONS · p. 29
  • Relative paths must begin with “./”, absolute paths must be full paths, and agents must use the initially given working directory.Examples distinguish relative paths for new files from absolute paths for data.Source: IMPORTANT STATE AND PATH INSTRUCTIONS · p. 29
  • Agents should check intermediate values against physical ranges and investigate values that are impossible or orders of magnitude off.Examples cover stellar masses, distances, velocities, and probabilities.Source: IMPORTANT STATE AND PATH INSTRUCTIONS · p. 29

FINAL INSTRUCTIONS

  • Agents must execute code for computation, include file_text when creating files, and submit only computed results in JSON, using None for failures and full-precision floats.The prompt also directs agents to install packages with python3 -m pip and save downloads and new files locally.Source: FINAL INSTRUCTIONS · p. 30

D.2 AUTOMATIC TASK EXTRACTION IN REPLICATIONBENCH-PLUS

  • The extraction prompt asks researchers to identify 4–6 key analysis tasks from a scientific paper: important numerical results reproducible through computation and implementable with reasonable resources, up to one node and a modest GPU.Source: §D.2 · p. 31, §2 · p. 31

TASK FORMAT

  • Each task record includes an identifier, kind, difficulty, brief description, and detailed implementation instructions with context and an exact output format.Task instructions must not refer to figures or other tasks.Source: TASK FORMAT · pp. 31–32
  • Task records also specify expected outputs, tolerances, output justifications, and prerequisite task IDs; the response is a JSON array of 4–6 task objects.The justification identifies the source of the expected output.Source: TASK FORMAT · p. 32
Loading 2510.24591v2…