NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

1. Introduction

• NatureBench tests discovery on genuine natural-science problems, unlike paper-based benchmarks focused on reimplementation [Siegel et al., 2024, Starace et al., 2025, Wang et al., 2026] and engineering-optimization benchmarks [Chan et al., 2025, Nathani et al., 2025, Qiang et al., 2026, Rank et al., 2026].The paper argues these existing benchmarks do not adequately test domain reasoning, specialized tooling, or cross-discipline knowledge, and can be fragile to rerun because of environment fragmentation.Source: §1 · p. 3
• NatureBench comprises 90 tasks from Nature-family papers across six scientific domains, with an information firewall that removes each source method to require discovery rather than reproduction.It uses a SOTA-normalized relative gap as its primary metric and a post-hoc validity judge to detect shortcut behavior.Source: §1 · p. 3
• NatureGym’s three-stage build-then-verify pipeline converts papers into reproducible containerized task packages with a task brief, dataset, held-out test set with hidden ground truth, and automated evaluator.The benchmark build starts from approximately 5,500 papers published in ten Nature-family journals between 2022 and 2025.Source: §1 · p. 3, Figure 2 · p. 4
• 17.8% of tasks were surpassed by Claude Opus 4.7 at g>0.1, while it matched published SOTA on 47.8%.The evaluation used a strict web-search-disabled protocol.Source: §1 · p. 3
• Methodological translation accounted for 45.5% of validated successes, while wrong method choice caused 45.1% of failures and insufficient compute budget 24.4%.In a ten-agent behavioral analysis of 900 task–agent runs, success primarily came from reframing scientific tasks as familiar supervised-prediction problems; failures were not primarily due to task misunderstanding.Source: §1 · p. 3

2. NatureGym

• NatureGym converts eligible Nature-family papers into containerized tasks with agent-visible data, held-out tests, automated evaluation, SOTA anchors, and a firewall withholding the source method.The package standardizes papers with heterogeneous formats, toolchains, and data modalities.Source: §2 · p. 4

2.1. Pipeline Overview

• Each task passes through paper filtering, dataset acquisition and verification, and package construction, with an independent review and verify–repair loop at every stage.Stages also make pass-or-reject decisions and accumulate structured task information for downstream construction.Source: §2.1 · p. 4

2.2. Paper Filtering

• Filtering requires an extractable ML task, deterministic automated evaluation of a quality-related SOTA metric, and complete, publicly accessible matching data; datasets over 50 GB are rejected.At least one evaluation instance must be feasible, with development data, test inputs, and reference answers.Source: §2.2 · p. 5

2.3. Dataset Acquisition and Verification

• Dataset acquisition checks downloaded files, retains method-independent task inputs, excludes algorithm-specific operations and outputs, and verifies evaluation splits and core-instance validity.A read-only review cross-checks the paper, code, and files; a repair step reconciles the task record and dataset before construction.Source: §2.3 · pp. 5–6

2.4. Task Package Construction

• Task construction separates agent-visible inputs from hidden references and scoring logic, documents task metrics and SOTA scores, and validates outputs with a reference-type-specific evaluator.The package is checked for evidence-grounded fidelity, firewall integrity, and executable consistency; automated checks include end-to-end testing.Source: §2.4 · pp. 6–7

3. NatureBench

• NatureBench combines paper-sourced genuine scientific problems with optimization-oriented evaluation against published SOTA, unlike benchmarks focused on reproduction or non-scientific optimization tasks.The benchmark contains 90 task packages across six scientific task domains.Source: §3 · pp. 7–8

3.1. Source Corpus

• The pipeline narrows approximately 5,500 candidates from ten Nature-family journals to 90 calibrated task packages, using public data availability and numerical SOTA claims to select source journals.The publication window is 2022–2025; the final set draws from six journals.Source: §3.1 · p. 8, Table 3 · p. 8

3.2. Benchmark Quality Calibration

• Claude Opus 4.6 reproduced 30 tasks and DeepSeek-V4-Pro reproduced 21 at g≥−0.05; on 16 shared successes, median g was −0.0026 and 90% of deviations were ≤0.031.Reproduction-mode auditing dropped 45 defective tasks and repaired 17, leaving 90 packages; remaining non-successes were attributed to resource limits or agent capability rather than package defects.Source: §3.2 · p. 9

3.3. Benchmark Statistics

• Across 90 tasks and 333 evaluation instances, NatureBench spans six scientific domains, eight ML task types, six journals, and varied source-paper contribution types.The corpus is concentrated in Nature Machine Intelligence (36 tasks), Nature Methods (26), and Nature Computational Science (16); task counts by publication year from 2022 to 2025 are 11, 17, 28, and 34.Source: §3.3 · pp. 9–10, Figure 3 · p. 9
• Tasks vary in data scale, modality, evaluation splits, answer paradigm, and metric, with 81 distinct primary metrics making raw scores across tasks incomparable.Tasks use a mean of 3.7 primary and 5.1 auxiliary metrics; evaluation may use hidden labels, distributional scores, or an oracle scorer.Source: §3.3 · pp. 10–11

3.4. Evaluation Protocol

• Each isolated task is scored by a direction-normalized, scale-free SOTA-relative gap on its designated primary metric, with g≥0 indicating a match or exceedance.Agents receive a four-hour budget in containers that hide the evaluator, ground truth, and SOTA target; a post-hoc judge flags shortcut behavior and invalidates flagged runs.Source: §3.4 · p. 12

4.1. Experimental Setup

• Twelve models paired with three CLI harnesses ran independently across all 90 tasks under a shared protocol, given visible task data and specifications to develop and submit solutions.Web search was disabled, and each task received a 4-hour budget and GPU resources matched to its recorded compute requirements.Source: §4.1 · pp. 12–13

4.2. Main Results

• Claude Opus 4.7 led overall with 17.8% Surpass-SOTA (g>0.1) and 47.8% Match-SOTA (g≥0), while clear improvements were uncommon across agents.Gemini 3.5 Flash and GLM-5.2 each reached 15.6% Surpass-SOTA; the best Match-SOTA rate remained below half.Source: Table 4 · p. 13, §4.2 · p. 13
• Claude Opus agents had 100% completion and score rates with no invalid submissions; GPT-5.5 had 13 invalid submissions but retained 44.4% Match-SOTA and a +0.001 median accepted-task gap.Invalid shortcuts were filtered from scores, so GPT-5.5’s results reflect judge-accepted submissions.Source: §4.2 · p. 13, Table 5 · p. 14
• Median relative gaps ranged from −0.007 for Claude Opus 4.7 to −0.40 for MiniMax-M2.7, with most scores modestly below SOTA and extreme shortfalls pulling means lower.The paper treats Surpass-SOTA and Match-SOTA as primary metrics because the SOTA-normalized gap can amplify large shortfalls.Source: §4.2 · p. 14, Table 5 · p. 14

5.3. Benchmark Validity

• Extreme relative-gap scores reflected the SOTA-relative metric’s behavior rather than task errors, so the analysis prioritizes Surpass- and Match-SOTA and uses median relative gap as a tail-robust summary.A near-ceiling SOTA can yield a large negative gap for a moderate agent; a large positive gap can occur when the scored primary metric captures only one facet of a multi-objective paper method.Source: Figure 6 · p. 15, §5.3 · p. 18

5.1. Solution Mechanisms

• 32.2% of runs across ten analyzed agents matched SOTA.The analysis attributes the gap primarily to method choice and execution depth, rather than task misunderstanding or malformed output.Source: §5.1 · p. 14, §5 · p. 14, Figure 7 · p. 16
• 41.4% of runs used supervised predictive modeling, while engineering-driven approaches accounted for 82.7% of successful runs.Runs using the same broad method family as the source paper matched SOTA in 37.7% of cases, versus 29.6% for runs using a different family.Source: §5.1 · p. 15, Figure 7 · p. 16
• Among below-SOTA or invalid runs, 61.1% were method-layer failures and 28.7% execution-layer failures; wrong method choice alone accounted for 45.1%.Insufficient budget or time accounted for 24.4% of failures; understanding-layer and strategy-layer failures accounted for 3.1% and 7.0%, respectively.Source: §5.1 · pp. 15–16, Figure 7 · p. 16

5.2. Domain and Interdisciplinary Variation

• Consensus Match-SOTA rates ranged from 60.0% in Relational Reasoning to 17.9% in Biomedical Modeling, with a difficulty ordering shared across agents.The easier tier also included Protein Biology (37.5%) and Cellular Omics (35.5%); the harder tier included Physical Modeling (26.9%) and Molecular Design (18.2%). All ten agents’ rankings correlated positively with the consensus ordering (Spearman ρ = 0.71–1.00).Source: §5.2 · p. 16, Figure 8 · p. 17
• The pooled median relative gap widened from −0.13 on single-discipline tasks to −0.21 on cross-discipline tasks, while Match-SOTA rates fell from 33.1% to 28.0%.The relative gap moved in this direction for 9 of 10 agents, and Match-SOTA rates were lower for 8 of 10.Source: Figure 8 · p. 17, §5.2 · p. 17

5.3. Benchmark Validity

• Tasks that cover only a bounded slice of a source paper retain its core quantitative problem, so Surpass- and Match-SOTA assess that slice rather than the whole paper.Omitted directions or instances may be infeasible to structure, score deterministically, or collect within data-volume budgets.Source: §5.3 · p. 18
• With web search disabled and a post-hoc validity judge filtering submissions, review found high-frequency submissions were overwhelmingly legitimate iteration and rare exploits were caught.The protocol constrains residual leakage and feedback risks from public data, input-target coupling, and repeated exact-score feedback.Source: §5.3 · p. 18

6.1. AI for Science

• AI-for-science systems have advanced prediction and discovery across fields, but humans generally define the research program, data, and success criterion, limiting evidence for cross-disciplinary problem solving.NatureBench tests this horizontal capability on 90 Nature-family tasks across six scientific domains, using each paper’s reported SOTA as its discovery scoring anchor.Source: §6.1 · pp. 18–19

6.2. Paper-based Benchmarks

• NatureBench shifts paper-based evaluation from understanding, review, reproduction, replication, or rediscovery to independently solving the source paper’s problem against its reported SOTA.PaperQA evaluates citation-backed answers [Asai et al., 2024, Lala et al., 2023, Skarlinski et al., 2024], while PaperBench tests paper reconstruction [Starace et al., 2025].Source: §6.2 · pp. 19–20

6.3. AI-train-AI and Autonomous Optimization

• Existing AI-train-AI and autonomous-optimization work evaluates ML experimentation suites, specialized verifier-driven discovery, or end-to-end research automation, leaving large-scale paper-grounded SOTA evaluation underexplored.NatureBench combines benchmark-suite scale, paper-sourced science, and SOTA-referenced evaluation across six scientific task domains.Source: §6.3 · p. 20

• NatureGym constructs per-task scientific environments from Nature-family papers, while NatureBench benchmarks 90 Nature-sourced tasks across six domains for reproduction and discovery.The authors release both resources alongside a public leaderboard with maintainer-side reproduction.Source: §7 · p. 20
• The strongest of ten frontier agents surpasses published SOTA (g>0.1) on only 17.8% of tasks and matches it on 47.8%.Source: §7 · p. 20
• Agents succeed mainly through methodological translation into familiar supervised-prediction problems, while failures stem chiefly from wrong method choice (45.1%) and insufficient compute budget (24.4%), not task misunderstanding.Source: §7 · p. 20

Package and Environment Review Details

• The review iterates through verification and repair until the artifact is complete, consistent, stably scorable, and buildable, while preserving the information firewall.Unresolved uncertainties and issues that cannot be reliably auto-repaired are escalated to human review.Source: A. Package and Environment Review… · pp. 29–30

Build-time self-audit

• Before construction ends, a self-audit rechecks the task definition, data alignment, metadata tags, SOTA scores, and information firewall against the paper and structured record.Anything uncertain is flagged for human review before the review loop proceeds.Source: A. Package and Environment Review… · p. 29

Task-package verification

• The task-package verification runs 36 checks across five dimensions, including artifact completeness, consistency, firewall compliance, benchmark design, and end-to-end dynamic testing.Dynamic tests use a baseline across all instances, check evaluator outputs, test near-perfect scoring for ground-truth predictions, and ensure malformed inputs fail cleanly; failed checks trigger targeted repairs followed by reruns.Source: A. Package and Environment Review… · p. 30

Environment verification

• Environment verification builds the Docker image on physical hardware, checks imports and preset library versions, then diagnoses and repairs build failures without overriding base-image packages.Repairs proceed from compatible versions and missing dependencies to alternatives or package removal; task-critical dependencies that resist these steps trigger a standalone Dockerfile, while evaluator and solver dependencies remain mandatory.Source: A. Package and Environment Review… · p. 30

B. Benchmark Quality Calibration Details

• First-round diagnosis identifies six defect classes, including ground-truth leakage, distorted task definitions, shortcut-insensitive metrics, evaluator or anchor inconsistencies, pipeline errors, and missing data.Locally verifiable defects receive targeted repairs; risks are backstopped by an information firewall, web-search-disabled container, and validity judge, while irreparable cases are dropped.Source: B. Benchmark Quality Calibration Details · p. 30
• The reproduction audit decomposes methods into components and checks package quality across task/data, scoring, metadata anchors, and cross-component consistency; 45 tasks are dropped and 17 receive minor repairs.Checks include anchor/evaluator dataset and scale alignment, resource availability, ground-truth score reasonableness, and consistency with the paper’s method.Source: B. Benchmark Quality Calibration Details · pp. 30–31
• Claude Opus 4.6 successfully reproduces 30/90 tasks and partially reproduces 16/90, versus DeepSeek-V4-Pro’s 21/90 and 13/90.Success is defined as g≥−0.05, while partial success is −0.2 ≤g<−0.05.Source: B. Benchmark Quality Calibration Details · p. 31
• At least one model succeeds on 35/90 tasks, and both succeed on 16/90; on jointly successful tasks, median g is −0.0026.For 90% of those joint successes, absolute deviations are at most 0.031.Source: B. Benchmark Quality Calibration Details · p. 31
• Reproduce-mode success is lower than base mode: Opus falls from 41/90 to 30/90 and DeepSeek from 29/90 to 21/90, primarily because faithful reproduction requires heavier training and complex dependencies.DeepSeek’s no-result count rises from 1 to 29; non-success is dominated by insufficient compute or time and method simplification rather than package defects.Source: B. Benchmark Quality Calibration Details · p. 31

Case 1: method-aligned graph modeling can produce a valid success

• Claude Opus 4.7 exceeded paper-side SOTA on the cancer-gene task with an appropriate graph neural network ensemble.Results varied by network: MTG, LTG, PCNet, and Multinet improved substantially, while IRef v15 remained slightly below SOTA; the agent did not invent a new identification method.Source: C. Case Studies · pp. 32–33

Case 2: extensive valid iteration can still fall short

• The agent produced 258 submissions, with its best score at attempt 220; this was a method-layer limitation, as its chosen models lacked the inductive bias and representation capacity of large-scale genomic pretraining.The failure was not a formatting or execution failure.Source: C. Case Studies · p. 33

Case 3: a plausible route can be limited by execution depth

• The reaction-prediction agent’s valid sequence-to-sequence approach was limited by execution depth: the fixed budget did not allow sufficiently deep training or efficient inference.The agent implemented a trained model with checkpointing and beam-search inference, but the task required a longer training and generation loop.Source: C. Case Studies · p. 34

D. Resource Usage Details

• Usage is aggregated from valid execution logs at the trajectory level, reporting per-case input tokens, output tokens, and estimated API cost.Input tokens use exact harness- or provider-reported fields; when cache accounting is available, non-cached, cache-read, and cache-creation tokens are retained separately, and the mean input-token count sums them.Source: D. Resource Usage Details · p. 34
• Incomplete third-party output-token logs are estimated from agent-authored trajectory text at roughly one token per four English characters [Anthropic, 2026c, OpenAI, 2026c].Claude Opus, GPT, and Gemini output counts use exact provider- or harness-reported fields; estimated rows and derived costs are marked with an asterisk.Source: D. Resource Usage Details · p. 34
• Table 11 reports per-agent mean token usage and estimated API costs over valid runs with recorded usage information.Costs use official standard list prices and provider-specific cache rates, excluding promotions, alternative service modes, regional uplifts, unrecoverable cache-storage charges, and unrecoverable OpenAI long-context multipliers; asterisked costs derive from estimated output tokens.Source: D. Resource Usage Details · p. 35, Table 11 · p. 35