PaperlayerView exact revision on arXiv

Source-linked AI summary

NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

Yuru Wang, Lejun Cheng, Yuxin Zuo, Sihang Zeng, Bingxiang He, Che Jiang, Junlin Yang, Yuchong Wang, Kaikai Zhao, Weifeng Huang, Kai Tian, Zhenzhao Yuan, Jincheng Zhong, Weizhi Wang, Ning Ding, Bowen Zhou, Kaiyan Zhang

arXiv:2606.24530v2cs.CL

TL;DR

Existing benchmarks do not establish whether coding agents can discover competitive methods for real, cross-disciplinary scientific problems. NatureBench and NatureGym evaluate this capability across 90 containerized tasks, finding that the strongest agent surpasses published SOTA on only 17.8% of tasks, with success driven mainly by methodological translation rather than scientific invention.

  • Problem

    Existing benchmarks largely test reproduction or engineering optimization, leaving limited evidence about whether agents can discover competitive methods for real, cross-disciplinary scientific problems.

  • Method

    NatureBench evaluates discovery across 90 Nature-sourced tasks using NatureGym’s standardized, containerized environments with hidden original methods and automated evaluation.

  • Results

    17.8% of tasks surpassed published SOTA, while agents’ dominant success pathway was methodological translation rather than scientific invention.

  • Takeaways & Limitations

    Agents primarily solve these scientific tasks by translating them into familiar supervised-prediction problems rather than inventing scientific methods.

  • Takeaways & Limitations

    The SOTA-relative metric can be distorted by near-ceiling SOTA values or by primary metrics that capture only one facet of multi-objective methods.

Abstract

from arXiv · show

We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench

1. Introduction

NatureBench evaluates whether AI coding agents can discover competitive scientific methods rather than merely reproduce published implementations. Built on NatureGym, it uses 90 Nature-family tasks and finds limited SOTA-matching performance under web-search-disabled evaluation, with success driven mainly by methodological translation.

  • Benchmark motivation and contribution: Existing paper-based benchmarks test re-implementation of published methods but do not determine whether agents can independently discover competitive methods.This limitation motivates evaluating discovery rather than reproduction.
  • Benchmark motivation and contribution: 90 tasks form NatureBench, a cross-discipline benchmark distilled from peer-reviewed Nature-family publications.The benchmark is designed to test whether agents can move beyond reproduction toward discovery.
  • Benchmark motivation and contribution: NatureGym converts published papers into reproducible, containerized per-task environments, addressing environment fragmentation in prior agent-on-research benchmarks.Its task packages include a task brief, the paper’s dataset, and a held-out test set with hidden ground truth.
  • Evaluation and findings: 17.8% of tasks were surpassed on published SOTA, while 47.8% were matched by the strongest agent, Claude Opus 4.7, under a strict web-search-disabled protocol.The reported SOTA criterion is 𝑔> 0.1.
  • Evaluation and findings: Success was driven primarily by methodological translation, in which agents converted scientific tasks into familiar supervised prediction problems.The behavioral analysis covered 900 task–agent runs.

2. NatureGym

NatureGym converts published Nature-family papers into reproducible, containerized agent tasks with standardized data, evaluation, and SOTA anchors. Its staged pipeline filters for feasible tasks, verifies datasets, and enforces an information firewall that withholds source methods from agents.

  • Task standardization: Each NatureGym task packages a brief, dataset, held-out test set, automated evaluator, and SOTA anchor score in a standardized container.The format accommodates heterogeneous papers, toolchains, and data modalities while supporting reproducible evaluation.
  • Pipeline stages: The pipeline proceeds through paper filtering, dataset acquisition and verification, and task package construction, with independent verify–repair reviews between stages.Each stage makes a pass-or-reject decision and refines a structured per-paper record for downstream construction.
  • Feasibility filtering: Filtering requires an extractable ML task, a deterministic fully automated quality evaluation, and complete publicly accessible data with development and evaluation splits.Papers are rejected when ML is auxiliary, evaluation depends on human judgment or external services, or data is incomplete or exceeds 50 GB.
  • Information firewall: The file-level firewall exposes task-defining inputs and method-agnostic resources while excluding algorithm-specific preprocessing, intermediate outputs, and final outputs.Agents begin at the core algorithm’s inputs, receiving none of its operations or outputs.
  • Package construction: Task construction separates agent-visible inputs from hidden reference answers and scoring logic, while documentation specifies data schemas, metrics, output formats, and submission requirements.Only quality metrics used for paper ranking are retained, with one primary metric designated per instance for aggregate scoring.

3. NatureBench

NatureBench is a 90-task benchmark spanning six scientific domains that evaluates optimization on paper-sourced scientific problems against published SOTA. Its construction combines NatureGym’s funnel and evaluation-time calibration to produce a heterogeneous, reproducible corpus.

  • Benchmark positioning: NatureBench contains 90 task packages spanning six scientific domains and combines paper-sourced scientific problems with optimization-oriented evaluation against published SOTA.It is presented as distinct from benchmarks focused on reproduction or non-scientific optimization tasks.
  • Source selection and construction: The NatureGym funnel narrows ∼5,500 candidates from ten Nature-family journals to ∼2,500 research articles, then ∼200 papers, ∼180 datasets, and the final task corpus.Journal selection requires concrete algorithmic contributions with numerical SOTA claims, publicly recoverable data, and scientific machine-learning scope.
  • Quality calibration: Evaluation-time calibration repairs locally verifiable defects, audits reproduction-mode support, and drops tasks with irreparable or systematic problems.Claude Opus 4.6 and DeepSeek-V4-Pro audited task descriptions, data, evaluators, metadata anchors, and cross-component consistency; 45 tasks were dropped and 17 received minor repairs.
  • Reproducibility calibration: 30 tasks were reproduced successfully by Claude Opus 4.6 and 21 by DeepSeek-V4-Pro under 𝑔≥−0.05, while the 16 jointly successful tasks had median 𝑔 −0.0026.90% of deviations were ≤0.031, supporting calibrated SOTA anchors; remaining non-successes were attributed to resource budget and agent capability rather than package defects.
  • Corpus composition and evaluation design: The final benchmark comprises 90 tasks and 333 evaluation instances distributed across six journals, six scientific domains, diverse ML task types, modalities, and source-paper contribution types.The corpus includes 81 distinct primary metrics and heterogeneous data-partition topologies, motivating a direction-normalized, scale-free relative-gap metric for cross-task comparison.

4. Experiments

NatureBench evaluates twelve coding-agent configurations under a shared, web-search-disabled protocol against the published SOTA targets of 90 tasks. Clear improvements are rare: Claude Opus 4.7 surpasses SOTA on 17.8% of tasks and matches it on 47.8%, while most scores remain modestly below SOTA.

  • Evaluation protocol: Agents autonomously develop and iteratively submit solutions using only each task’s visible data and problem specification, scored against its source paper’s published SOTA target.The evaluation uses a single shared protocol across NatureBench tasks.
  • Models: 12 models are evaluated through three CLI-based agent harnesses: Claude Code, Codex CLI, and Gemini CLI.The configurations include nine models with Claude Code, two with Codex CLI, and Gemini CLI pairings as specified in the experiments.
  • Overall performance: 47.8% is Claude Opus 4.7’s Match-SOTA rate, the highest result, followed by GPT-5.5 at 44.4%, GLM-5.2 at 41.1%, and Gemini 3.5 Flash at 37.8%.Match-SOTA uses g≥0, and even the leading agent remains below half of the 90 tasks.
  • Overall performance: 17.8% is Claude Opus 4.7’s Surpass-SOTA rate, the highest clear-improvement result; Gemini 3.5 Flash and GLM-5.2 each reach 15.6%, while MiniMax-M2.7 reaches 1.1%.Surpass-SOTA uses g>0.1.
  • Score distribution: The median relative gap ˜g_all ranges from −0.007 for Claude Opus 4.7 to −0.40 for MiniMax-M2.7, indicating that most tasks remain modestly below SOTA.Weaker agents shift more scores toward severe failure, while only a minority of tasks per agent reach SOTA.

5. Analysis

Across 900 runs from ten agents, performance gaps to published SOTA arise mainly from method choice and execution depth: agents typically translate scientific tasks into generic supervised-learning pipelines rather than discovering domain-informed methods. Performance also varies systematically by domain and disciplinary scope, while metric normalization, task coverage, and leakage introduce bounded interpretive limitations.

  • Solution mechanisms: 32.2% of runs matched published SOTA, with agents mainly failing on method choice and execution depth rather than task misunderstanding.The analysis annotated 900 runs across 90 tasks and ten agents.
  • Method pathways: 41.4% of runs used supervised predictive modeling, reflecting agents’ systematic shift from paper methods toward familiar generic ML families.Paper-side methods concentrated in structured representation, statistical modeling, and pretraining or transfer learning.
  • Success modes: 82.7% of successful runs relied on engineering-driven strategies, led by supervised proxy prediction at 45.5%.Other engineering-driven success modes were optimization and tuning at 17.6%, engineering pipelines at 11.0%, and pretraining or model scaling at 8.6%.
  • Failure modes: 61.1% of below-SOTA or invalid runs were method-layer failures, chiefly wrong method choice at 45.1%, while execution-layer failures accounted for 28.7%.Insufficient budget or time accounted for 24.4% of these runs; understanding-layer and strategy-layer failures accounted for 3.1% and 7.0%.
  • Domain and disciplinary scope: Consensus Match-SOTA rates across domains ranged from 60.0% to 17.9%, with cross-discipline tasks widening the gap from a pooled median relative gap of −0.13 to −0.21.The domain ordering was consistent across agents, with Spearman ρ ranging from 0.71 to 1.00.
  • Robustness and limitations: The SOTA-relative metric, bounded task coverage, and residual leakage or feedback risks constrain interpretation without invalidating the benchmark design.Extreme negative gaps can result from near-ceiling SOTA denominators, while some tasks capture only a scorable subset of the source paper.

6. Related Work

Prior AI-for-science systems largely operate within human-defined research programs, whereas newer scientific agents aim to choose methods and execute experiments as primary problem solvers. NatureBench preserves paper grounding but evaluates independent solution of the paper’s scientific problem against its reported SOTA, including across disciplines.

  • AI-assisted research: Human researchers typically define programs, curate data, and set success criteria, with AI serving as an instrument within the research program.This paradigm is characterized as a revolution of tools rather than a tool of revolution.
  • AI-native problem solving: General-purpose scientific agents increasingly aim to choose methods, run experiments, and optimize for final scientific outcomes.The cited examples include The AI Scientist, the AI co-scientist, DeepScientist, and AutoSOTA.
  • Cross-disciplinary evaluation: Cross-disciplinary evaluation tests whether AI-native solvers can break scientists’ information cocoon by combining methods, data, and concepts across fields.The motivation is that specialized training, expanding literature, and field-specific tooling impede cross-disciplinary integration.
  • Paper-based benchmarks: Paper-based benchmarks assess understanding, review, reproduction, replication, reproducibility assessment, or rediscovery of known findings rather than independently solving the source paper’s problem.Examples include PaperQA, OpenScholar, LAB-Bench, ReviewerGPT, PaperBench, AutoExperiment, and LMR-Bench.
  • NatureBench: NatureBench retains paper grounding while shifting evaluation to independent solution of the same scientific problem, using reported SOTA as the scoring anchor.The target is to match or surpass the source paper’s reported SOTA.

7. Conclusion

NatureGym standardizes per-task scientific environments for NatureBench, a 90-task benchmark spanning six domains that evaluates reproduction and discovery. Across ten frontier agents, the strongest surpasses published SOTA on 17.8% of tasks and matches it on 47.8%, with success driven mainly by methodological translation rather than scientific invention.

  • Contributions: NatureGym constructs per-task scientific environments from Nature-family papers, enabling NatureBench to evaluate reproduction and discovery across 90 tasks in six scientific domains.The benchmark uses these standardized environments to address scientific tasks across disciplines.
  • Results: 17.8% of tasks: the strongest of ten frontier agents surpasses published SOTA under the g>0.1 criterion.The same agent matches published SOTA on 47.8% of tasks.
  • Results: Agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised-prediction problems rather than demonstrating scientific invention.This finding characterizes the dominant pathway underlying successful agent performance.

8. Authors

The paper is authored by Yuru Wang, Lejun Cheng, Yuxin Zuo, Ning Ding, Bowen Zhou, and Kaiyan Zhang, affiliated with Horizon Research at Frontis.AI, Tsinghua University, Peking University, and Harvard University.

  • The authors are Yuru Wang, Lejun Cheng, Yuxin Zuo, Ning Ding, Bowen Zhou, and Kaiyan Zhang.
  • The listed affiliations include Horizon Research at Frontis.AI, Tsinghua University, Peking University, and Harvard University.

A. Package and Environment Review Details

The package and environment review uses an iterative verify–repair loop to produce structurally complete, internally consistent, stably scorable, and buildable artifacts while preserving the information firewall. It combines build-time self-audit, 36 task-package checks, and Docker-based environment verification with minimally disruptive repairs.

  • Review workflow: The verify–repair loop iterates until artifacts are structurally complete, internally consistent, stably scorable, and buildable while preserving the information firewall.The review differs from one-shot filtering and data-acquisition reviews by repeatedly checking and repairing the final artifact.
  • Build-time self-audit: A final build-time self-audit rechecks task definitions, data alignment, metadata tags, SOTA scores, and the firewall, flagging uncertainties for human review.The audit rereads both the paper and the structured record before construction is completed.
  • Task-package verification: 36 checks cover artifact completeness, cross-component consistency, the information firewall, benchmark-design principles, and end-to-end dynamic testing.Dynamic testing runs a simple README-following baseline solver over all instances, feeds outputs to the evaluator, and checks score structure and values.
  • Environment verification: Environment verification builds the Docker image on a physical machine, tests imports, matches library versions to presets, and repairs failures without overriding base-image packages.Repairs proceed from least to most disruptive: use a base-compatible version, add missing dependencies or runtime configuration, then substitute a compatible alternative.

B. Benchmark Quality Calibration Details

Benchmark calibration identifies six defect categories, filters irreparable problems, and retains legitimate low-scoring or invalid runs as agent failures. Reproduction audits decompose methods and assess both outcome attribution and package quality, while finalized-task results quantify SOTA-anchor attainability and the cost of faithful reproduction.

  • Calibration diagnosis: Six diagnosis categories include ground-truth leakage, distorted task definitions, shortcut-insensitive metrics, and evaluator or anchor inconsistencies.The appendix states that first-round exposed defects fall into six categories, with the remaining categories not fully shown in the supplied passage.
  • Calibration safeguards: Tasks with broken definitions, unverifiable metrics, or irreparable leakage are dropped, while legitimate low-scoring, timed-out, or invalid runs remain agent failures.The filtering process is backstopped by the information firewall, web-search-disabled container, and validity judge.
  • Reproduction audit: Reproduction audits decompose each paper method into components, classify score outcomes, attribute anomalies, and assess package quality across four axes.The audit covers preprocessing, architecture, loss, training, inference, and post-processing, plus task/data, evaluator/scoring, metadata anchors, and corrections or environment fixes.
  • Reproducibility analysis: 30/90 tasks were successfully reproduced by Claude Opus 4.6 and 21/90 by DeepSeek-V4-Pro, while at least one model succeeded on 35/90.The success criterion was g≥−0.05, with partial success defined as −0.2 ≤g< −0.05; both models succeeded on 16/90 tasks.
  • Reproducibility analysis: Reproduce-mode success fell from 41/90 to 30/90 for Opus and from 29/90 to 21/90 for DeepSeek, primarily because faithful reproduction required heavier training and more complex dependencies.For tasks where both models succeeded, g clustered tightly around zero, with median −0.0026 and 90% of absolute deviations ≤0.031.

C. Case Studies

The case studies span three recurring outcomes: a method-aligned solution exceeding SOTA, a valid but methodologically insufficient solution, and a plausible route limited by execution depth. Together, they show that agent failures can arise from inadequate method choice or computational depth rather than invalid execution.

  • Case 1: Method-aligned success: The cancer-gene agent correctly used graph-based node classification with an appropriate GNN, imbalance handling, early stopping, and ensembling to exceed SOTA.MTG, LTG, PCNet, and Multinet improved substantially, while IRef v15 remained slightly below paper-side SOTA; the agent did not propose a new identification method.
  • Case 2: Methodological insufficiency: 258 submissions culminated in the best genomic-sequence score at attempt 220 after extensive iteration from k-mer models to CNNs, ensembles, classifiers, and threshold sweeps.The judge marked the submission valid because all predictions were generated by models trained on the provided data.
  • Case 2: Methodological insufficiency: The genomic-sequence failure was methodological, not formatting or execution-related: the models lacked the inductive bias and representation capacity of large-scale genomic pretraining.This trajectory exemplifies failures that are runnable but not strong enough.
  • Case 3: Execution-depth limitation: The reaction-prediction agent implemented a valid sequence-to-sequence route, but the fixed budget prevented the deeper training, efficient generation, and specialized chemical modeling required.The case illustrates execution-layer failure: a plausible direction can fail when the training and inference loop is too long.

D. Resource Usage Details

Resource usage is measured at the trajectory level, aggregating per-case input tokens, output tokens, and estimated API cost from valid execution logs. Token accounting uses provider-reported fields when available and transparent estimates otherwise, while costs apply standard list prices with unrecoverable or excluded adjustments omitted.

  • Measurement Protocol: Resource usage is aggregated per agent and case from valid execution logs, covering input tokens, output tokens, and estimated API cost.Input tokens use exact harness- or provider-reported usage fields, preserving non-cached, cache-read, and cache-hit distinctions when logged.
  • Measurement Protocol: Output tokens use exact provider- or harness-reported fields for Claude Opus, GPT, and Gemini runs.For third-party models executed through Claude Code, incomplete logged fields are replaced by estimates from agent-authored trajectory text; affected rows and costs are marked with an asterisk.
  • Cost Accounting: Costs are computed using official standard list prices and provider-specific cache rates.Limited-time promotions, alternative service modes, regional uplifts, unrecoverable cache-storage charges, and unrecoverable OpenAI long-context multipliers are excluded.
  • Reported Statistics: Table 11 reports per-agent means over valid runs with recorded usage information.Asterisks identify values estimated from agent-authored trajectory text and costs derived from those estimates.
Loading 2606.24530v2…