PAPERBENCHX: Can AI Reproduce Science Across Disciplines?

1 Introduction

• Outside machine learning, reproducing a result can require domain-specific physical and numerical choices, and a successful run may still model the wrong physical system.The environment is part of the scientific problem, not merely infrastructure.Source: §1 · p. 2
• Plausible outputs are insufficient: reproduction requires a valid workflow whose evidence can be independently regenerated to support the scientific claim.Agents must reconstruct experiments, execute and diagnose them, and verify that the resulting evidence supports the target claim.Source: §1 · pp. 2–3
• PAPERBENCHX comprises 93 expert-constructed paper-reproduction tasks across 12 research areas and six scientific domains, using 10 domain-native computing environments.Agents receive the paper, a public task specification, and the required environment, but not the private reference workflow, evidence, or rubric.Source: §1 · p. 3
• Across 10 configurations, the strongest exceeds the 90% full-reproduction threshold on only 13 tasks, while mean rubric scores range from 27.53% to 59.08%.Agents often make substantial progress in modeling and execution but fail at scientific validation.Source: §1 · p. 3

2.1 Machine-Learning Research and Paper Reproduction

• Machine-learning benchmarks evaluate experimentation or paper-level reproduction, but remain centered on relatively homogeneous software stacks.MLAgentBench and RE-Bench target task-level metric improvement; PaperBench and SA-Bench use reproduction or semantic-alignment rubrics (Starace et al., 2025; Hu et al., 2026).Source: §2.1 · p. 3

2.2 Cross-Domain Scientific Benchmarks

• Cross-domain benchmarks generally isolate parts of reproduction, such as scientific coding, discovery, data analysis, released-code reruns, or repository repair.ScienceIDE evaluates domain-native execution, but its tasks concern repository repair and implementation rather than reproducing published experiments.Source: §2.2 · p. 4
• PAPERBENCHX combines paper-level reproduction, heterogeneous domain-native environments, and replay-grounded verification of regenerated evidence.A plausible answer or completed repository is not sufficient; submitted workflows must replay from scratch and support the target claim.Source: §2.2 · p. 4

3.1 Task Definition

• Each task supplies a published paper, a public specification of scope and deliverables, and a domain-native environment that is part of the task definition.The agent submits a runnable package; during clean replay, the evaluator restores that package and runs its canonical entry point.Source: §3.1 · p. 4

3.2 Task Statistics

• The 93 tasks span 12 research areas, 10 scientific environments, and 3,168 scorable criteria across 93 source papers and 44 venues.Task time limits range from 4 to 72 hours, with a median of 7 hours; Figure 3 depicts task, venue, and rubric coverage.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 Task Construction Pipeline

• Experts select a reproducible claim, define its scope and evidence requirements, provision the environment, write the public contract, and verify an initial reference run.A candidate task advances only after the scoped workflow executes successfully within the available resources.Source: §3.3 · p. 6
• Admission checks include rubric construction, clean reference replay, calibration on successful and invalid workflows, and independent expert review.Failed tasks return for rebuilding or rubric revision, and downstream checks are repeated; construction and validation took more than 1,000 expert hours.Source: §3.3 · pp. 6–7

3.4 Evaluation

• Submissions are clean-replayed and only regenerated artifacts are admissible evidence; grading combines deterministic checks with domain interpretation against the hidden rubric.Rubric leaves assess Modeling, Execution, or Validation; partial score is the mean task score, while full rate counts tasks scoring strictly above 90%.Source: §3.4 · p. 7

4.1 Experimental Setup

• The evaluation used ten model–scaffold configurations with the same public materials and shell access, without access to private reference workflows or rubrics.All configurations were evaluated on the same 93 tasks; submissions were clean-replayed and graded by the same GPT-5.6-sol evaluator with deterministic checks.Source: §4.1 · p. 8

4.2 Main Results

• 59.08% partial score and 13.98% full rate were achieved by GPT-6-Astra (Codex), which exceeded the 90% threshold on only 13 of 93 tasks.Across models, macro-average partial score was 49.26% versus an average full rate of 8.60%; Validation averaged 42.91%, below Modeling at 62.73% and Execution at 61.83%.Source: §4.2 · p. 8, Table 2 · p. 8
• Mean partial scores ranged from 34.05% for RF filters and frequency-selective surfaces to 84.32% for computational materials science.Figure 5 presents scores by model and research area, domain difficulty, and overall scores.Source: §4.2 · p. 8, Figure 5 · p. 9
• GPT-6-Astra led six of the 12 research areas, while Fable 5 and Fable 5.1 each led three; no configuration dominated every area.Source: §4.2 · p. 9

• Only 80 of 930 outcomes exceeded 90%; successes covered 22 tasks, while no configuration completed 71 tasks.Eleven of the remaining 22 tasks were completed by at most two configurations.Source: §5 · p. 9
• Validation’s median was 40.0%, below Modeling’s 73.4% and Execution’s 73.5%.Figure 6(a) presents the stage-score distributions across 930 outcomes.Source: §5 · p. 9, Figure 6 · p. 10
• Of 304 outcomes scoring at least 80% in both Modeling and Execution, only 53 (17.4%) exceeded the full-reproduction threshold.Their mean Validation score was 58.37%.Source: §5 · p. 9
• Interaction turns had only weak negative correlation with Partial score (r = −0.14); longer trajectories did not ensure better scores.Fable 5.1 and Fable 5 averaged about 113 turns with Partial scores of 58.17% and 57.96%, while GPT-6-Astra scored 59.08% at 348.7 turns.Source: §5 · p. 9, Figure 6 · p. 10
• Native execution consumed 51.8–63.3% of tool-mediated time, but models differed in how effectively they converted it into valid evidence.Fable 5 devoted 37.1% to paper reconstruction and implementation, the largest upstream share among the four sampled models.Source: §5 · p. 9, Figure 6 · p. 10
• On the F-stub UWB MIMO task, Fable 5 scored 28.9% in 159 turns and recovered evidence for three antenna variants.GPT-6-Astra scored 27.4% in 464 turns and HY3 scored 18.1% in 390 turns; the case illustrates how reconstruction, parameter choices, and workflow design shape verifiable evidence.Source: §5 · p. 10
• Some workflows execute but support incorrect conclusions; others invalidate evidence through caching, privileged test-state information, or premature termination.Separate Modeling, Execution, and Validation scores plus replay and provenance checks distinguish these failure modes.Source: §5 · p. 10

• The best configuration achieved a 59.08% Partial score but only a 13.98% full-reproduction rate; no configuration fully reproduced 71 tasks.Source: §6 · p. 10
• The evaluation covers 10 frontier agent configurations across 93 tasks in 12 research areas and 10 domain-native environments.Source: §6 · p. 10
• Agents often built plausible models and ran native simulations, but validation remained their weakest stage, and more interaction did not necessarily improve performance.Source: §6 · p. 10
• The results frame scientific reproduction as an evidence-grounded checkpoint for AI scientists and motivate better diagnosis, provenance, and verification.Source: §6 · p. 10

A.1 Anatomy and Information Boundary of a Complete Task

• The GMR task asks agents to reproduce guided-mode resonances in a three-pair multilayer dielectric grating using Meep.The starter submission and scientific environment are agent-facing during a controlled trial; tests, rubric, judge prompt, and reference artifacts remain evaluator-side.Source: §A.1 · p. 15
• The task directory includes its specification, environment setup, starter workflow, source paper materials, and tests and evaluation files.The listed files include task.toml, instruction.md, Docker configuration, reproduce.sh, paper materials, test.sh, evaluate.py, and rubric.json.Source: §A.1 · p. 15
• Table A2 condenses the scientific requirements that determine what must be executed and what evidence retained, but does not replace the complete protocol.Source: §A.1 · p. 15, Table A2 · p. 16
• The hidden rubric checks independent Modeling, Execution, and Validation leaves, including geometry, clean Meep replay artifacts, power closure, pole fits, mode agreement, and convergence.A missing item affects dependent leaves rather than imposing a global zero.Source: §A.1 · p. 15
• Only evidence regenerated by clean replay is admissible; the evaluator accepts documented data layouts and unit conversions but does not infer missing scientific quantities or provenance.Scientific failures score zero on affected leaves, while evaluator read or tool failures withhold judgment for repair and reevaluation.Source: §A.1 · p. 16
• After evaluation, the pinned public demonstration directory contains the full instruction, scientific addendum, environment, judge prompt, rubric, and reference materials.Source: §A.1 · p. 16

A.2 Complete Benchmark Inventory

• Table A3 is the complete 93-task result-set inventory; the excerpt lists examples from photonics, chemistry, materials science, biology, and robotics.Source: Table A3 · p. 16, §A.2 · pp. 16–17

A.3 Public Demonstration Tasks

• The public release includes twelve demonstration tasks and their source publications, with identifiers retained to map directly to released task directories.The directory mapping is pinned to commit 7abb21c4.Source: Table A4 · p. 18, §A.3 · p. 18

A.4 Platforms and Resource Envelopes

• Table A5 lists the execution stacks and declared resource limits for the twelve public task packages; Figure 7 shows representative solver, analysis, and embodied-task outputs.Source: Table A5 · p. 18, Figure 7 · p. 19

B.1 Scope, Resources, and Trial Records

• 93 tasks and 3,168 rubric leaves are scored across a fixed denominator; Full rate requires an aggregate task score strictly greater than 0.9.Agent time limits range from 4 to 72 hours, with a median of 7 hours.Source: §B.1 · p. 19

B.2 Expert Annotation and Release Review

• Domain experts define each task’s claim, configuration, conditions, observables, alternatives, and required evidence, then validate feasibility and scoring before release.Two independent experts review scope, fairness, reproducibility, evidence requirements, and scoring behavior.Source: §B.2 · pp. 19–20

B.3 Clean Replay and Evidence Grading

• Clean replay removes generated outputs and mutable state, runs the submitted package under controlled conditions, and freezes only replay-regenerated artifacts as evidence.Deterministic checks and expert-evaluated rubric leaves score the evidence; execution records alone do not establish scientific correctness.Source: §B.3 · p. 20

B.4 Regression and Adversarial Checks

• Regression tests pair valid controls with corrupted or incomplete submissions to check evidence admission, localize failures, and resist prompt injection.The verifier checks include numerical parsing, native-solver evidence, provenance, cached-output removal, and evaluator-owned records.Source: §B.4 · p. 20

C Additional Result Diagnostics

• Table A6 summarizes verifier regression cases, pairing valid controls with invalid fixtures so permissible variation is not mistaken for unsupported evidence.Source: Table A6 · p. 21

C.1 Task Completion

• 71 of 93 tasks have no configuration above the operational 90% full-reproduction threshold.Figure 8 reports each task’s maximum partial score and the number of configurations exceeding 90%.Source: §C.1 · p. 21, Figure 8 · p. 21

C.2 Cross-Simulator Trace Case Studies

• 43.5% for GPT-6-Astra and 41.3% for Opus 5 on smartphone MIMO, but neither reproduced the central four-port matching target or efficiency conditions.Both ran native HFSS workflows and recovered several secondary criteria; DeepSeek-V4.1-Flash scored 15.3% after 1,024 turns.Source: §C.2 · p. 22
• 28.9% for Fable 5, 27.4% for GPT-6-Astra, and 18.1% for HY3 on F-stub UWB MIMO; none reproduced the principal S-parameter response.All three completed native variants and recovered some secondary evidence, while the comparison shows more interactions do not necessarily improve reproduction when modeling remains incorrect.Source: §C.2 · p. 22
• 73.1% for Fable 5.1 and 72.9% for Fable 5 on the LNOI coupler task, despite full Execution credit for three configurations.Scientific errors remained in overlap, facet-PDL, and EME coupling criteria; Kimi-K3’s EME output was NaN despite an 83.6% Modeling score.Source: §C.2 · p. 22
• Modeling, Execution, and Validation are reported separately because the cases diagnose different stages rather than establish a cross-task ranking.Figure 9 previews the stage scores and aggregate scores for the three cases.Source: §C.2 · pp. 21–22, Figure 9 · p. 22