Source-linked AI summary
PaperBench: Evaluating AI's Ability to Replicate AI Research
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, Tejal Patwardhan
TL;DR
AI agents need to reproduce ML research from scratch, but scalable and objective evaluation of such long-horizon work remains difficult. PaperBench combines author-co-developed hierarchical rubrics, automated judging, and experiments across 20 papers; the strongest tested agent scores 21.0%, below the human baseline on a subset.
Problem
AI agents’ ability to autonomously replicate state-of-the-art ML research requires evaluation of understanding, coding, experimentation, and long-horizon execution.
Method
PaperBench evaluates 20 ICML papers with 8,316 author-co-developed rubric outcomes and uses an LLM-based judge assessed through JudgeEval.
Results
21.0% is the average Replication Score achieved by Claude 3.5 Sonnet (New), while ML PhDs achieved 41.4% versus o1’s 26.6% on a three-paper subset.
Takeaways & Limitations
Current AI agents show non-trivial ability on some replication facets but remain far from competently performing the full range of tasks required for successful replication.
Takeaways & Limitations
The LLM-based judge is less accurate than an expert human and non-deterministic because it uses non-deterministic model calls.
Abstract
from arXiv · showhide
We introduce PaperBench, a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch, including understanding paper contributions, developing a codebase, and successfully executing experiments. For objective evaluation, we develop rubrics that hierarchically decompose each replication task into smaller sub-tasks with clear grading criteria. In total, PaperBench contains 8,316 individually gradable tasks. Rubrics are co-developed with the author(s) of each ICML paper for accuracy and realism. To enable scalable evaluation, we also develop an LLM-based judge to automatically grade replication attempts against rubrics, and assess our judge's performance by creating a separate benchmark for judges. We evaluate several frontier models on PaperBench, finding that the best-performing tested agent, Claude 3.5 Sonnet (New) with open-source scaffolding, achieves an average replication score of 21.0%. Finally, we recruit top ML PhDs to attempt a subset of PaperBench, finding that models do not yet outperform the human baseline. We open-source our code (https://github.com/openai/preparedness) to facilitate future research in understanding the AI engineering capabilities of AI agents.
1. Introduction
PaperBench evaluates whether AI agents can replicate ML research from scratch, using author-informed rubrics and automated judging. Frontier-agent results show substantial but limited replication ability.
- Benchmark motivation: PaperBench asks agents to understand papers, build codebases from scratch, and run, monitor, and troubleshoot experiments.Complete replication is described as highly challenging, requiring human experts several days at minimum.
- Benchmark design: 20 ICML 2024 Spotlight and Oral papers span 12 topics and contain 8,316 individually gradable outcomes.Each paper has a manually created rubric co-developed with an original paper author.
- Automated evaluation: o3-mini-high with custom scaffolding achieves an F1 score of 0.83 as an LLM-based judge on JudgeEval.JudgeEval compares automated-judge outputs with human expert gold labels.
- Benchmark design: Hierarchical rubrics decompose replication into fine-grained sub-outcomes, enabling granular measurement of partial progress.The benchmark’s samples pair research papers with grading criteria for complete replication.
- Results: 21.0% is the PaperBench score achieved by Claude 3.5 Sonnet (New) with a simple agentic scaffold.On a three-paper subset, ML PhDs reached 41.4% after 48 hours, versus 26.6% for o1.
- Contributions: PaperBench includes a benchmark, a lighter Code-Dev variant, JudgeEval, and frontier-model evaluations for long-horizon ML R&D.These contributions support studying agents’ ability to conduct complex research tasks.
2. PaperBench
PaperBench evaluates from-scratch research replication through clean execution and weighted hierarchical grading. It also offers a less robust Code-Dev variant that grades implementation without reproduction.
- 2.1. Task: Candidates submit repositories with a reproduce.sh entrypoint that must reproduce empirical results from a fresh setup.Agents do not see the rubric and cannot use authors’ original codebases.
- 2.2. Reproduction: A separate reproduction phase runs submissions cleanly, distinguishing generated outputs from results hard-coded during task time.Execution produces results, plots, and a reproduce.log file.
- 2.3. Grading: Leaf requirements receive binary scores, and weighted child averages propagate to the root Replication Score.The score is a weight-adjusted proportion of satisfied rubric requirements, with 100% representing all leaf requirements satisfied.
- 2.3. Grading: The main metric is the average Replication Score across all papers.Submissions without reproduce.sh receive zero on Execution and Result Match nodes.
- 2.4. Requirement Types: PaperBench grades Code Development, Execution, and Result Match nodes to award credit for implementation, running code, and reproduced results.Execution and Code Development nodes provide partial progress toward matching results.
- 2.6. PaperBench Code-Dev: PaperBench Code-Dev skips reproduction and grades only Code Development nodes to reduce inference and compute costs.It is more accessible but less robust, and its performance is only weakly correlated with the full evaluation.
3. Dataset
The dataset samples contemporary ML research through 20 ICML 2024 papers and author-collaborated hierarchical rubrics. These rubrics provide thousands of weighted, granular requirements for partial-progress measurement.
- 3. Dataset: PaperBench contains 20 ICML 2024 Spotlight and Oral papers curated to represent contemporary AI research.The collection spans topics including reinforcement learning, robustness, and probabilistic methods.
- Rubric construction: Each paper’s rubric was written with an original author and required multiple weeks for reading, review, iteration, and sign-off.The authors’ collaboration was intended to ensure rubric quality and accuracy.
- Rubric construction: Rubrics decompose core outcomes into increasingly fine-grained child requirements, with leaf completion sufficient to assess overall success.The rubric tree begins with the highest-level replication outcome and decomposes core contributions into specific outcomes.
- Rubric construction: PaperBench has 8,316 leaf nodes, each granular enough for an expert familiar with the paper to review in under 15 minutes.Granularity enables scoring partial attempts and simplifies judging individual requirements.
- Rubric construction: Rubric-node weights encode the importance of contributions relative to sibling nodes rather than implementation difficulty.The weighting rewards prioritizing more important paper components.
- Rubric construction: Paper-specific addendums record author clarifications, out-of-scope components, and, when needed, judge-only reference information.These addendums supplement each paper’s rubric.
4. LLM Judge
Because human grading takes tens of hours per paper, PaperBench introduces SimpleJudge and JudgeEval to make rubric-based evaluation scalable and assess automated-judge accuracy. The selected setup, o3-mini with SimpleJudge scaffolding, achieves an F1 score of 0.83 at approximately $66 per paper.
- Motivation: Tens of hours per paper made expert grading impractical, motivating an automated evaluation method.A single PaperBench rubric typically contains hundreds of nodes to evaluate.
- SimpleJudge: SimpleJudge independently grades each rubric leaf node using the paper, rubric, node requirement, and submission.When the submission is too long, the judge ranks files by relevance and includes only the top ten files.
- Scalability: The LLM judge is significantly cheaper and faster than expert-human grading, while judge quality is expected to improve over time.For PaperBench Code-Dev, the estimated grading cost drops to around $10 USD per paper.
- JudgeEval: JudgeEval evaluates automated judges against human-graded leaf nodes from partial replications of five papers.The replications were created from scratch or by modifying original author codebases.
- JudgeEval: 0.83 F1 at $66 USD per paper makes o3-mini with SimpleJudge scaffolding the selected judge setup for the main results.The comparison uses macro-averaging across papers and includes GPT-4o, o1-mini, o1, and o3-mini judge models.
5. Experiments and Results
PaperBench evaluates frontier agents on from-scratch replication under constrained execution, compares them with ML PhD attempts, and tests how scaffolding and time affect performance. Agents show non-trivial but limited long-horizon replication ability, with human performance eventually surpassing extended o1 performance.
- Setup: Agents receive paper materials and autonomously implement, execute, monitor, and troubleshoot complete replications in an Ubuntu container with one A10 GPU.The environment provides internet access, package downloads, and API services, while the main run is capped at 12 hours.
- Main results: Claude 3.5 Sonnet scores 21.0% on average, compared with 13.2% for o1 and under 10% for the other tested models.These are average Replication Scores in the BasicAgent main setup.
- Failure modes: Models frequently finish early, fail to strategize within the time limit, and struggle to execute long sequences of planned actions.The authors identify these behaviors as weaknesses in long-horizon task execution; o3-mini also frequently struggles with tool usage.
- Scaffolding: IterativeAgent significantly boosts o1 and o3-mini relative to BasicAgent but hampers Claude 3.5 Sonnet, showing strong sensitivity to prompting.The variant removes early termination and prompts piecemeal work; the paper reports that Claude’s BasicAgent advantage over o1 reverses under IterativeAgent.
- Long-horizon performance: o1 initially outperforms humans during an extended run but falls behind after 24 hours, with scores mostly plateauing after the first hour.The comparison uses hourly snapshots from a 36-hour o1 IterativeAgent run and human performance over time.
6. Related Work
Related benchmarks evaluate research engineering, competition solving, toy research, or automated judging, while PaperBench targets from-scratch replication of modern ML papers across broader, longer-horizon requirements.
- Research engineering benchmarks: CORE-Bench gives agents a paper repository, whereas PaperBench requires reproducing paper results from scratch.The distinction isolates research replication without relying on an existing repository.
- Competition benchmarks: MLE-bench, MLAgentBench, and DSBench focus on Kaggle competitions, while PaperBench contains tasks drawn from contemporary machine learning research.The paper characterizes many Kaggle challenges as dated and relatively simple compared with its target tasks.
- Open-ended research tasks: Compared with RE-Bench’s seven open-ended tasks, PaperBench covers more sub-tasks over a longer work horizon and emphasizes performance across complex replication requirements.RE-Bench generally provides a scoring function, whereas PaperBench uses rubric-based evaluation across paper replication tasks.
- Research capability studies: Prior studies report LLM research ideas comparable in novelty to human PhDs in specific domains and success on some toy research problems.Those toy problems include hypothesis formation, experiment design and execution, and result analysis.
- Automated judging: PaperBench benchmarks automated judges on harder tasks than earlier work and evaluates them against human-graded labels through JudgeEval.The paper also notes prior evidence that agent-based judges can outperform non-agent judges on certain tasks.
7. Limitations
PaperBench’s scope, contamination risk, rubric-construction burden, judge reliability, and evaluation cost constrain how its results should be interpreted and extended.
- Dataset size: The benchmark currently contains only 20 papers, limiting coverage of the broader ML research community despite thousands of rubric-level requirements.Each rubric contains hundreds of nodes, so the paper count understates the number of evaluated requirements.
- Contamination: Online author codebases may enable contamination for future models, even though current models are considered unlikely to be affected because the papers are recent.The benchmark disallows agents from viewing original codebases, but pretraining may still internalize solutions.
- Dataset creation: Creating detailed, accurate rubrics requires several full days of expert work and is difficult to reproduce at the desired quality.The authors report that each rubric requires deep paper understanding and careful review, and that training others to create them is challenging.
- Judge performance: The LLM-based judge is less accurate than an expert human judge and non-deterministic because it relies on non-deterministic model calls.The authors call for further work and adversarial stress-testing of automated judges.
- Cost: A 12-hour o1 IterativeAgent rollout costs about $400 per paper, while grading with o3-mini SimpleJudge adds approximately $66 per paper.PaperBench Code-Dev is designed to reduce GPU and cost requirements; its grading cost averages $10 per paper, and an experimental judge shows preliminary 10x lower grading costs.
8. Conclusion
PaperBench is a demanding benchmark for testing whether AI agents can replicate cutting-edge machine learning research through complete code-and-experiment workflows. Results show meaningful but limited current capability, while the benchmark supports ongoing evaluation of AI autonomy in ML research.
- PaperBench evaluates agents on building codebases from scratch, conducting complex experiments, and generating final research results.
- Claude 3.5 Sonnet (New) achieved a 21.0% average Replication Score, indicating substantial difficulty with the full replication process.
- Current agents can replicate certain facets of ML papers but remain far from competently performing the full range of tasks required for successful replication.
- Open-sourcing PaperBench provides a method for evaluating, monitoring, and forecasting autonomous AI R&D capabilities, while covering only one part of the broader evaluation landscape.
Impact Statement
Autonomous replication of frontier research could accelerate scientific and AI safety progress, but it could also make innovation outpace understanding and safeguards. PaperBench is presented as one measure for tracking these emerging capabilities.
- Autonomous replication of cutting-edge ML research may accelerate scientific discovery, including AI safety and alignment research.
- Greater autonomy in replicating and extending frontier research could produce innovation faster than risk assessment, governance, and safety interventions can keep pace.
- PaperBench is intended to measure emerging autonomous R&D capabilities while representing only one part of the broader evaluation landscape.
A. Future Directions in AI Evaluation
The paper identifies challenges in making complex, unstructured research evaluation scalable and trustworthy. Future directions include better rubric dependencies and automation, while dataset construction and grading impose important scope and cost constraints.
- A. Future Directions in AI Evaluation: PaperBench converts complex, underspecified replication into smaller rubric-based criteria, with paper authors helping address ambiguity.
- A. Future Directions in AI Evaluation: Current rubrics encode dependencies through child-node order but do not specify exactly which earlier requirements each later requirement needs.
- A. Future Directions in AI Evaluation: Frontier models can help summarize papers but struggle to create reliable rubrics end to end, motivating human-in-the-loop workflows and model-based rubric critique.
- A. Future Directions in AI Evaluation: Human grading is prohibitively expensive because a single rubric typically contains hundreds of nodes and can take tens of hours to evaluate.
- A. Future Directions in AI Evaluation: More reliable judges may allow less fine-grained task decomposition, but the trade-off between specification and delegation remains future work.
- A. Future Directions in AI Evaluation: The benchmark excludes or filters papers with inaccessible dependencies, distributed-training requirements, closed-source model dependencies, human data collection, insufficient reproducibility detail, or primarily framework-focused contributions.
G. More on JudgeEval
JudgeEval examines the accuracy and cost of automated rubric grading, finding that model-based judges can approach human performance at lower cost. Pruning rubrics can further reduce cost, but results remain preliminary and grading PaperBench is still expensive.
- JudgeEval compares automated judges with human-graded labels using performance and API-based cost per paper, alongside human and random baselines.
- Humans cost hundreds of dollars more per paper than the most expensive evaluated model, o1.
- o3-mini achieves performance comparable to o1 at one-tenth the cost.
- Judge performance varies by requirement type: Code Development is hardest, while Result Match is easiest; o3-mini-high reaches 0.72 F1 on Code Development.
- PaperBench grading still costs around $66 per paper on average with o3-mini, and Code-Dev omits Execution and Result Match tasks.
- Pruning rubric trees beyond depth 3 reduced grading cost by 10× in one submission while causing only slight performance deterioration.
- Pruning results are preliminary, with observed cases of unsatisfactory judge performance.
I.1. Results stratified by requirement type
Models score better on Code Development than on Execution and Result Match requirements, indicating a gap between producing code and using it to obtain reproduced results. PaperBench therefore evaluates execution and result matching separately for a more holistic assessment.
- Results by requirement type: Models perform better on Code Development nodes than on Execution and Result Match nodes.The authors interpret this pattern as models writing substantial code but struggling to integrate, test, and run it successfully.
- Results by requirement type: Submissions receive a 0% result when disqualified for violating PaperBench rules.The reported model result tables mark such outcomes with an asterisk.