From Scientific Observations to Mechanisms: Benchmarking Hypothesis Generation by AI Scientists

1 INTRODUCTION

• A mechanistic hypothesis is an open-form account of how processes and their interactions produce an observed phenomenon, including the conditions under which they operate.In a catalytic-reaction example, molecules blocking active sites could explain a slowdown over time.Source: §1 · p. 1
• Existing benchmarks often test pattern or equation recovery rather than how underlying processes produce phenomena under specific conditions.Real datasets rarely identify a unique, fully specified mechanism, while synthetic data simplify observations; prior benchmarks do not fully combine empirical data with a controllable mechanism.Source: §1 · p. 2
• Open-form hypotheses can express equivalent mechanisms differently and imply outcomes only under some conditions, so wording or single-label scoring is insufficient.The evaluation instead tests outcomes implied under unseen conditions and supports agents that iteratively inspect data and execute analyses.Source: §1 · p. 2
• MechHypoBench combines paper-derived mechanisms from 14 scientific fields with independently collected empirical datasets containing 17.98 million observation records.The datasets retain observational complexity while the specified mechanisms provide ground truth.Source: §1 · p. 2
• Experiments reveal a substantial gap between agent-generated hypotheses and the underlying mechanisms.The benchmark evaluates open-form hypotheses through their implied consequences under withheld changes in conditions.Source: §1 · pp. 2–3

2 RELATED WORK

• Data-driven benchmarks recover structured targets such as relationships, equations, enumerable hypothesis spaces, or mechanistic models, while other work explores open-form generation.The paper distinguishes these lines from literature- and knowledge-driven idea or hypothesis generation.Source: §2 · p. 3
• Existing empirical settings generally lack explicit grounding in fully specified mechanisms, while controlled synthetic settings simplify real scientific observations.MechHypoBench evaluates open-form mechanistic hypothesis generation from empirical datasets while preserving observational complexity.Source: §2 · p. 3
• Table 1 compares scientific-discovery benchmarks by scientific fields and empirical-data coverage, among other dimensions.Its caption defines partial coverage and identifies the number of scientific subjects and use of empirical scientific datasets.Source: Table 1 · p. 3

3.1 CONCEPT FORMALIZATION

• A condition combines driver, context, history, and phase; a mechanistic hypothesis is an open-form account of processes and conditions producing observed phenomena.The observations pair conditions with mechanism-generated responses, while multiple hypotheses may explain them to differing degrees.Source: §3.1 · p. 4

3.2 SCIENTIFIC SOURCE COLLECTION

• The collection includes 306 reviewed papers across 14 scientific fields and 200 independently collected datasets with manually annotated experimental roles and diverse structures.Paper claims were manually checked against source papers; datasets underwent provenance, reuse, and structural review.Source: §3.2 · pp. 4–5

3.4 TASK CONSTRUCTION

• Each task fixes a paper-derived mechanism, aligns it with a dataset through experimental roles, and gives agents public observations while withholding matched comparisons for assessment.Comparisons differ in one condition component and have stable ground-truth response-change directions; task difficulty reflects errors of a marginal baseline.Source: §3.4 · pp. 5–6, Figure 3 · p. 5

3.5 HUMAN EVALUATION

• 4.90 for probe clarity and 4.96–4.97 for reference-answer validity; scientific coherence scored 4.60–4.62 and mechanism-task consistency 4.54–4.69 after revisions.Human review prompted reconstruction of 17 paper-derived mechanisms with insufficient scientific coherence.Source: Table 2 · p. 6, §3.5 · p. 6

3.6 EVALUATION FRAMEWORK

• A separate interpreter receives only each hypothesis and comparison descriptions, then returns probabilities for decrease, no change, or increase; higher MCA and lower MCNLL indicate closer mechanism behavior.MCA measures directional agreement, while MCNLL penalizes confident errors more strongly.Source: §3.6 · p. 6

4.1 EXPERIMENTAL SETUP

• The experiments evaluate 300 tasks (120 easy, 180 hard) using six general-agent configurations with and without skills, plus AI-Scientist-v2 and AutoDiscovery.AutoDiscovery is evaluated on a fixed, stratified subset of 40 tasks because its search averaged 111.6 accepted LLM calls per task.Source: §4.1 · p. 7

4.2 EVALUATING THE ABILITY OF GENERATING DATA-DRIVEN MECHANISTIC HYPOTHESIS

• Agents score lower MCA and higher MCNLL than QMR/PMR; skilled Codex with GPT-5.6 Sol leads agents but trails QMR by 22.7/32.1 MCA points and +0.468/+0.785 MCNLL (easy/hard).Empty and shuffled controls also perform substantially worse than mechanism references.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 EVALUATING HYPOTHESIS GENERATION QUALITY AND EFFICIENCY ACROSS HARNESSES AND SKILLS

• With backend and skill setting fixed, Claude Code generally achieves higher MCA than Codex; for DeepSeek-V4-Flash, both MCA and MCNLL favor Claude Code, which also makes more tool calls.Claude Code takes longer than Codex with DeepSeek-V4-Flash under both skill settings; its GLM-5.2 runtime is slightly longer without skills and faster with skills.Source: §4.3 · p. 8
• Scientific skills reduce mean execution time by 10.1–18.1% across four DeepSeek/GLM configurations, but MCA and MCNLL gains are limited or inconsistent.The runtime decrease does not hold for every model: Codex with GPT-5.6 Sol takes longer with skills, while Claude Code with Claude Sonnet 5 shows little change.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 EVALUATING MECHANISTIC HYPOTHESIS GENERATION IN AI SCIENTISTS

• AI-Scientist-v2 reaches 59.2%/43.8% MCA on easy/hard tasks, below Codex; AutoDiscovery reaches 27.5%/27.1%, versus 47.1%/35.8% for Codex without skills.AutoDiscovery has higher MCNLL probability for the ground-truth direction on the hard subset despite its lower MCA.Source: §4.4 · p. 8

4.5 EVALUATING EFFECTS OF EVIDENCE ACCESS AND RETENTION

• Evidence summaries raise MCA 10.0 points in both configurations (MCNLL reductions: 0.288 for Codex; 0.244 for Claude), while structured-record gains are 1.9 points for Codex and 2.1 for Claude, without consistent MCNLL improvement.For Codex, structured records improve implication of unchanged responses but reduce accuracy for increases or decreases.Source: §4.5 · p. 9

• MechHypoBench combines paper-derived mechanisms from 14 fields with independently collected empirical datasets.Source: §5 · p. 9
• Its evaluation framework assesses open-form hypotheses through their consequences under withheld conditions, using the paper-derived mechanism that generated the observations as ground truth.Source: §5 · p. 9
• Systematic experiments with AI agents identify limitations in mechanistic hypothesis generation and examine how evidence availability and retention affect their hypotheses.Source: §5 · p. 9