PaperlayerView exact revision on arXiv

Source-linked AI summary

EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights

Jiayi Geng, Zhengxuan Wu, Kevin S. Chen, Seungone Kim, Joseph Janssen, Zora Zhiruo Wang, Bhupalee Kalita, Runtian Gao, Aaron Ho, Andrew Oakleigh Nelson, Olexandr Isayev, Francisco Villaescusa-Navarro, Ching-Yao Lai, Howard Chen, Graham Neubig

arXiv:2610.00492v1cs.CLcs.AI

TL;DR

EUREKABENCH addresses whether AI agents can move beyond solution optimization to discover scientific insights. It evaluates agents conducting iterative experiments to uncover mechanisms, and finds that predictive accuracy can approach human performance while scientific-insight scores remain substantially lower.

  • Problem

    Whether AI agents can move beyond solution optimization and discover novel scientific insights remains an open question.

    Source: §1 · p. 2

  • Method

    EUREKABENCH evaluates agents using iterative experimentation to discover mechanisms, assessing scientific constraints, test-set predictive accuracy, and expert-formulated scientific insights.

    Source: §1 · pp. 2–3

  • Results

    GPT 6 Astra achieves 47.4% predictive accuracy versus 48.8% for humans, but its scientific-insight score is 29.4% versus 69.7% for human-discovered mechanisms.

    Source: §1 · p. 3

  • Takeaways & Limitations

    EUREKABENCH opens a path for studying AI-driven mechanism discovery across scientific fields, while whether agents can independently uncover meaningful insights remains unclear.

    Source: §1 · p. 3, §8 · p. 9

    The scientific-insight rubric captures only some possible discovery paths, so findings outside its questions do not count toward the score.

    Source: §8 · p. 9

Abstract

from arXiv · show

When Isaac Newton discovered the law of gravitation, he did so through an iterative process of analyzing observed data such as planetary patterns, finding the underlying mechanisms by describing patterns in mathematical equations, and refining his theory against the Moon's orbit, revealing the startling insight that the same force governs both falling apples and orbiting planets. Would it be possible for AI agents to make similar discoveries? To measure this ability, we introduce EurekaBench, a cross-domain benchmark that tests AI agents' ability to conduct long-horizon experiments and discover mechanisms that explain observations. We evaluate these mechanisms by the scientific insights that can be derived from them. EurekaBench contains an expert-verified set of 26 long-horizon tasks across neuroscience, computer science, chemistry, astrophysics, geophysics, and plasma physics, with a total of 306 scientific insights that the discovered mechanisms are expected to support. Our evaluation framework tests three axes of scientific discovery: agents' ability to follow known scientific constraints, the predictive accuracy of the discovered mechanisms, and whether these mechanisms yield scientific insights or inform future research. Our results show that current AI agents often overly fixate on predictive accuracy optimization, surpassing human scientists, while falling substantially short in deriving scientific insights.

1 Introduction

EUREKABENCH asks whether AI agents can move beyond optimizing solutions to discover mechanisms that yield scientific insights. It provides expert-crafted scientific problems and an iterative experimentation setting to evaluate that ability.

1 Introduction

  • EUREKABENCH comprises 26 expert-crafted scientific problems across six domains, developed in collaboration with 10 experts after consultation with 15 scientists.The domains are neuroscience, geophysics, astrophysics, computer science, plasma physics, and chemistry.Source: §1 · p. 2
  • The benchmark starts agents from observations that challenge existing understanding and asks them to use iterative experiments to discover understandable mechanisms that yield insights.Agents work from task specifications and a documented environment interface, then submit a mechanism description, implementation, and experimental records.Source: §1 · p. 2, Figure 3 · p. 3
  • The evaluation tests domain-specific scientific constraints, predictive accuracy on test measurements, and whether mechanisms support expert-formulated scientific insights.An agentic judge with access to the environment interface evaluates the discovered mechanisms.Source: §1 · pp. 2–3
  • GPT 6 Astra approaches human predictive accuracy (47.4% versus 48.8%) but achieves a 29.4% scientific-insight score, versus 69.7% for human-discovered mechanisms.The results indicate that fitting observed data for predictive accuracy is not enough to obtain mechanisms that deepen scientific understanding.Source: §1 · p. 3
  • Agents often treat mechanism discovery as task optimization; unresolved uncertainty and limited, in-depth exploration constrain their ability to discover insights.They show some scientific reasoning, such as establishing mechanisms behind physical parameters, but do not reliably carry out experiments to reduce uncertainty.Source: §1 · p. 3

2 Evaluating a Task in EUREKABENCH

EUREKABENCH tasks move from challenging initial observations through iterative experiments to a submitted mechanism, which is assessed for scientific validity, predictive accuracy, and derivable insights. Expert-built insight questions and agentic judges operationalize the evaluation.

2.1 Task Overview

  • A task is T = (x, o1, E, R): its specification and initial observations feed an experimental environment, while a rubric-based reward scores the deliverables.The agent submits an explanation and executable mechanism, an experiment log, and a trajectory of observations and actions.Source: §2.1 · pp. 3–4
  • Agents iteratively query the environment using their experiment history, receive observations, and revise hypotheses before submitting their mechanism and experiment record.The agent submits M, L, and τ for evaluation; the judge checks scientific constraints, test-set predictions, and whether insights can be derived from M.Source: §2.1 · pp. 3–4

2.2 Evaluation Protocol

  • Scientific constraint tests check domain common-sense assumptions, and a mechanism passes only when it satisfies every constraint.Predictive accuracy is measured by executing the mechanism on held-out test data; normalized criterion scores are summed and averaged.Source: §2.2 · p. 4
  • Scientific insight questions test whether each insight can be derived from the submitted mechanism, with expert-verified questions based on prior findings, open questions, and expert proposals.A perfect mechanism is expected to answer all questions; judges can query the environment and must provide complete evidence for a Yes verdict.Source: §2.2 · p. 4
  • The conditioned score combines predictive-accuracy and insight points only for mechanisms that pass all scientific-constraint tests.The protocol averages the two judges’ conditioned scores for each submission.Source: §2.2 · p. 4
  • Figure 4’s nine Antarctic ice-shelf insight questions concern differing deformation under compression and extension, directional resistance, and measurements that determine viscosity.The questions illustrate the kinds of scientific insights evaluated for a domain-specific task.Source: Figure 4 · p. 5

3 EUREKABENCH

EUREKABENCH was developed with domain experts around important but partially solved scientific problems, using task specifications, experimental environments, and expert-refined evaluation rubrics. Its 26 tasks across six domains include 306 insight questions.

3.1 Benchmark Construction

  • Experts with field PhDs helped identify important problems and potential insights, then refined task specifications and evaluation criteria through repeated discussion.The benchmark targets mechanism discovery as a valuable area for AI contribution to research.Source: §3.1 · p. 5
  • The selected problems are scientifically important but only partially solved, so insight questions assess what mechanisms can explain without requiring a complete ground-truth solution.Questions draw on existing human findings and potential insights proposed by domain experts.Source: §3.1 · p. 5
  • Each task asks agents to use iterative experimentation to find an understandable mechanism for initial observations that challenged experts’ understanding.Earlier observations are included as scientific background.Source: §3.1 · p. 5
  • The benchmark pairs each problem with an experimental environment and uses original-study or task-simulator data for predictive-accuracy testing.Environment dependencies are packaged in Apptainer images and accessed through command-line interfaces.Source: §3.1 · p. 5

3.2 Benchmark Statistics

  • EUREKABENCH contains 26 problems across six domains and 306 scientific insight questions for evaluating insights uncovered through mechanism discovery.The problems comprise 10 neuroscience, 2 geophysics, 3 plasma physics, 7 astrophysics, 3 computer science, and 1 chemistry task.Source: §3.2 · p. 5

4 Experiments

The experiments evaluate seven language models as autonomous agents, using provider-associated harnesses for closed-source models and a shared SDK for open-weight models. Runs have fixed compute budgets and restricted web access, with human progress serving as the baseline.

Models

  • Seven LLMs, including frontier closed-source and open-weight models, are evaluated on EUREKABENCH.The listed models include Claude Fable 5.1, GPT 6 Astra, GPT 5.6 Sol, DeepSeek V4 Flash, and Kimi K3; all run at maximum reasoning effort.Source: §4 · pp. 5–6

Experiments

  • Figure 5 presents the distribution of benchmark problems across domains, and Table 1 lists the environment interfaces.Source: Figure 5 · p. 6, Table 1 · p. 6

Agent harnesses

  • Claude and GPT models use their providers’ harnesses through OpenHands, while open-weight models use the OpenHands Agent SDK.The SDK supports different LLMs through a single interface.Source: §4 · p. 6

Compute budgets for AI agents

  • Each problem run gives an agent one H100 GPU, 1,000 iterations, and four hours to discover a mechanism through environment interaction.The human baseline is the current progress made by human scientists on each problem.Source: §4 · p. 6

Web access

  • Web search is allowed but access to the underlying research and literature aggregators is blocked, though pretraining exposure to source research is not ruled out.These restrictions apply to both search and webpage retrieval.Source: §4 · p. 6

5 Results

EUREKABENCH results show that agents struggle to discover mechanisms that explain observations and satisfy scientific constraints. Predictive accuracy approaches the human reference, but scientific insight scores remain substantially lower.

AI agents still struggle to find the underlying mechanism

  • 29.8% conditioned score for Claude Fable 5.1 compares with 63.7% for human-discovered mechanisms; AI-agent scores range from 12.9% to 29.8%.The results describe the conditioned score as indicating how well mechanisms explain the initial observations.Source: §5 · p. 6, Table 2 · p. 6
  • Except GPT 6 Astra (10/26), all agents fail scientific-constraint tests on more than half of EUREKABENCH problems.These tests assess whether mechanisms satisfy basic common-sense assumptions.Source: §5 · pp. 6–7

Agentic ability to discover scientific insights

  • 47.4% predictive accuracy for Claude Fable 5.1 and GPT 6 Astra, versus 48.8% for human solutions, contrasts with SI scores of 42.4%, 29.4%, and 69.7%, respectively.The main results table reports predictive accuracy (PA) and scientific insights (SI) unconditionally.Source: Table 2 · p. 6, §5 · p. 7
  • AI agents prioritize brute-force numerical optimization, while human scientists value mechanism interpretation and extrapolation; predictive accuracy alone does not imply novel insights.The result passages characterize agents’ focus as treating mechanism discovery as an optimization task.Source: §5 · p. 7

6 Analysis

The analysis tests whether guidance changes scientific insight scores and examines experiment logs for reasons agents fall short. Explicit insight questions improve scores, but expert reviews also identify failures to test assumptions, resolve uncertainty, and explore mechanisms further.

Effects of goals and insight questions on mechanism discovery

  • ∆+32.5% and ∆+15.8% SI contribution with +insights, versus ∆+10.0% and ∆+7.9% with +goal, on neural dynamics and ARC tokamak, respectively.The study evaluates Claude Fable 5.1; the authors note that explicit insight questions are not a natural setting because scientists do not yet know what insights a mechanism will yield.Source: §6 · p. 7, Figure 6 · p. 7
  • The +insights condition directly makes insight questions optimization targets, unlike general guidance to seek mechanisms useful for future exploration.The +goal condition increases SI contribution on both studied problems, but by less than +insights.Source: §6 · p. 7

Insights during experimentation and from discovered mechanisms

  • Figure 7 compares insights found during experimentation with insights derived from the mechanism on the same problems and instances.The caption specifies the shared problems and instances, not the direction or size of any difference.Source: Figure 7 · p. 7

Qualitative analysis: where do agents’ experiments fall short?

  • Experts identify four recurring failures: unjustified choices, equating fit with explanation, leaving uncertainty unresolved, and stopping before further exploration.Examples include unverified neural readouts, a possible numerical artifact left untested, and an unexplored turbulence-regime change.Source: §6 · pp. 7–8, Figure 8 · p. 8
  • Agents sometimes reject mechanisms that contradict evidence and recognize the physical meaning of variables, but their open-ended experimentation remains limited.The authors contrast these sound decisions with agents’ greater strength in task optimization than in resolving scientific uncertainty through experimentation.Source: §6 · p. 8, Figure 8 · p. 8

7 Related Work

Prior work studies scientific workflows, tool use, replication, and discovery benchmarks, including interactive settings and explanatory evaluation. EUREKABENCH distinguishes its focus as whether mechanisms yield scientifically meaningful insights during problem-solving, rather than merely a correct answer.

AI scientist agents

  • AI scientist agents span literature review, idea generation, experiments, paper writing, and mechanism revision across physical and computational systems.Related work also includes paper review, research memory, execution feedback, laboratory experimentation, and automated search for algorithms or equations.Source: §7 · p. 8

Benchmarking scientific discovery

  • Existing benchmarks assess research workflows, verifiable outputs, software engineering, experiment replication, numerical baselines, and hypotheses or equations recovered from data.Other interactive benchmarks let agents gather observations to uncover hidden systems and physical laws.Source: §7 · pp. 8–9
  • Unlike benchmarks centered on task completion or numerical performance, EUREKABENCH asks whether agents uncover scientifically meaningful insights while seeking an answer.DiscoverPhysics is described as a close setting that evaluates both trajectory prediction and conceptual explanations against expert rubrics.Source: §7 · p. 9

8 Conclusion

The conclusion leaves open whether AI agents can understand mechanisms and uncover novel insights, despite their strength in predictive optimization. It identifies limitations in the insight rubric and proposes experimentation-focused training as a possible path forward.

Conclusion

  • Agents often rely on untested assumptions, confuse task optimization with mechanism discovery, leave uncertainty unresolved, and stop without further exploration.The conclusion says mechanisms may also fail basic common-sense assumptions on many problems.Source: §8 · p. 9

Multiple paths in scientific discovery

  • The insight questions capture only some expert-selected research directions, so discoveries beyond the rubric do not count toward SI scores.The authors suggest expanding the insight set and having domain experts assess findings beyond established knowledge.Source: §8 · p. 9

Evaluating scientific insights

  • SI evaluation depends on judge agents’ scientific reasoning and may overestimate insights when judges accept confidently stated conclusions without establishing their derivation.Expert reviews offer broad judgments of selected submissions, not scores for every insight question.Source: §8 · p. 9

Training agents to discover scientific insights

  • Reinforcement-learning environments could reward agents for testing mechanisms and supporting insights with experimental evidence.The proposed training may help agents select informative experiments and investigate implications beyond initial observations.Source: §8 · p. 9

A Task Specification Template

The appendix identifies a neuroscience task-specification template and gives the agent a time budget and temporary directory for notes and intermediate files.

A Task Specification Template

B Guidance Analysis Prompts

The appendix presents prompt guidance for a goal condition and example instructions for neuroscience constraint and insight evaluation.

B.1 Goal Prompt

C.1 Scientific Constraint Evaluation

  • Table 5 gives example neuroscience instructions for scientific constraint tests.Source: Table 5 · p. 19

Judge instructions for insights derived from the mechanism

  • Table 6 gives neuroscience judge instructions for assessing insights derivable from a mechanism.The judge returns Yes only when complete evidence shows how the insight follows from the discovered mechanism.Source: Table 6 · p. 20

Judge instructions for insights discovered during experimentation

  • Table 7 gives example neuroscience judge instructions for insights discovered during experimentation.Source: Table 7 · p. 21

D Additional Results

This section presents additional findings on how predictive accuracy and scientific-insight scores relate, including recurring failures and useful experimental practices. It also reports neural response patterns and informative observations, and describes simulation interfaces used across four domains.

D.1 How do predictive accuracy and scientific insights vary across domains?

  • 67.1%–69.8% domain-average PA for the three Claude and two GPT agents in astrophysics accompanies SI scores of 33.6%–63.4%, while neuroscience and computer science scores are relatively low on both measures.Agents achieve higher PA in geophysics, plasma physics, and chemistry, but SI often remains much lower.Source: §D.1 · p. 22, Figure 9 · p. 22

D.2 Does higher accuracy imply more insights?

  • τb = 0.20 (adjusted p = 0.008), and the higher-PA agent scores lower on SI in 39.9% of untied comparisons, so accuracy alone is an unreliable indicator of insight.The reversal rate remains similar when either judge’s scores are used separately.Source: §D.2 · p. 22, Table 8 · p. 22

D.3 Common failures of AI scientist agents

Prediction with limited explanation (negative)

Treating task optimization as mechanism discovery (negative)

Interesting observations without further exploration (negative)

Using controls to rule out alternative explanations (positive)

  • A matched-voltage control isolated a conductance mechanism: added inhibition with compensating excitatory strength reduced the calcium first harmonic 35.92-fold, and the executable predicted 34.60-fold suppression.The same report retained both the successful gain prediction and a negative active-window R² in the balanced condition.Source: Using controls to rule out alternative… · p. 43, Figure 20 · p. 43, §D.4 · p. 43

Testing which model better predicts the observations (positive)

Revising models after experimental feedback (positive)

  • Within a stretch, block means were P1 0.10, P2 0.31, P3 0.24, and Q 0.17; the second verbatim presentation evoked a higher response than the first.The results report no repetition suppression within a stretch.Source: Revising models after experimental… · p. 45

Informative observations (positive)

  • Agents identify that measurements can vary independently, interventions can shift response timing without changing shape, and inactive processes leave some parameters poorly constrained.For c302, the proposed mechanism stores sensory perturbation as phase displacement along a command-supported recurrent motor–interneuron oscillation, rather than persistent sensory voltage or slowly clearing calcium.Source: Revising models after experimental… · p. 46, Informative observations (positive) · p. 46

E.1 Neuroscience

  • The neuroscience environments provide agents with c302 circuit interventions and simulations or TRIBE v2 predictions of stimulus-evoked fMRI responses (d’Ascoli et al., 2026).TRIBE v2 returns activity at 20,484 cortical locations; the c302 interface supports current injection, connection removal, and parameter changes.Source: §E.1 · p. 47

E.2 Geophysics

  • Geophysics tasks use FEniCSx for modifiable ice-shelf flow solvers and HBV catchment simulations driven by CAMELS-US data (Bergstr¨om, 1992; Addor et al., 2017).The HBV ground truth uses known values for 15 parameters, with noise added to simulated streamflow observations while retaining the generating parameters and noise-free streamflow.Source: §E.2 · p. 47

E.3 Plasma Physics

  • Plasma tasks use TORAX for transport simulations, while the negative-triangularity edge environment couples it with TokaMaker equilibrium and GPEC stability calculations (Citrin et al., 2024).Agents can vary plasma shape and operating conditions and inspect resulting equilibria and profiles.Source: §E.3 · p. 47

E.4 Astrophysics

  • Astrophysics tasks provide CLI access to CAMELS simulation suites and DREAMS Project runs for comparing cosmological parameters, feedback models, and dark matter scenarios (Rose et al., 2025).Available outputs include catalogs and, depending on the suite, particle snapshots, matter power spectra, or merger trees.Source: §E.4 · p. 47

E.5 Computer Science

The appendix describes computer-science and chemistry experiment interfaces, checks AIMNet2-rxn against published DFT calculations, and catalogs benchmark problems and sources.

E.5 Computer Science

  • The computer-science interfaces use LoRA fine-tuning (Hu et al., 2022) to test training-example effects and TransformerLens interventions to test candidate circuits.For belief-shift and subliminal-learning tasks, the agent can remove examples and repeat training; for circuit discovery, it replaces selected component activations and measures output changes.Source: §E.5 · p. 48

E.6 Chemistry

  • AIMNet2-rxn supplies energies and forces for H, C, N, and O neutral closed-shell structures, supporting geometry optimization and transition-state searches.Vibrational analysis tests for saddle points, and intrinsic reaction coordinate calculations test whether paths connect the intended reactants and products; the resulting paths and energy differences support identifying elementary steps, activation barriers, and reaction energies.Source: §E.6 · p. 48
  • AIMNet2-rxn retained 29 of 30 reference transition states as saddle points, while reactant-minimum barriers were 1.9–8.6 kcal/mol higher than reference.For 16 aldol transition states, relative energies differed by 1.85 kcal/mol on average and 3.51 at most; rankings correlated at 0.79, with both methods identifying the same lowest-energy state. For lactide, AIMNet2-rxn favored the hydrogen-bonded route over the covalent route by 13.9 kcal/mol, compared with 11.7 kcal/mol in the reference; differing DFT approximations mean discrepancies cannot be attributed entirely to AIMNet2-rxn.Source: §E.6 · p. 48

F Problem overview

  • Table 9 catalogs EUREKABENCH problems and their sources.Source: Table 9 · p. 49

Related papers

Loading 2610.00492v1…