PaperlayerView exact revision on arXiv

Source-linked AI summary

MechBench: Can AI Scientific Agents Discover Mechanisms Beyond Phenomenal Laws?

Zihan Yu, Jiadong Zhang, Jialin Cheng, Jingtao Ding, Yong Li

arXiv:2609.35515v1cs.AIcs.LGcs.SC

TL;DR

Scientific discovery benchmarks chiefly test whether agents recover observable laws, leaving reconstruction of the mechanisms generating them less directly evaluated. MechBench uses controlled mechanism variants, probes of internal consequences, and screening for ambiguous competitors; across agents, mechanism recovery trails law recovery, with the gap widening for more mutated mechanisms.

  • Problem

    Scientific-agent benchmarks primarily evaluate recovery of observable laws, while whether agents can reconstruct the mechanisms generating them remains difficult to test under controlled conditions.

    Source: §1 · p. 2

  • Method

    MechBench evaluates structured mechanisms through internal-consequence probes, scientifically interpretable mutations of canonical mechanisms, and screening for competing mechanisms that make recovery ambiguous.

    Source: §1 · p. 2

  • Results

    Across representative agents, mechanism recovery trails phenomenal-law recovery and the gap widens with mutation; Codex with GPT-5.6-sol scores 35.00% law accuracy and 13.75% mechanism accuracy on the Core-set.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Mechanism discovery is a distinct challenge beyond observable-law recovery, and remains difficult even when the correct phenomenal law is provided.

    Source: §6 · p. 9

    MechBench currently covers mechanisms whose observable laws can be obtained through tractable symbolic elimination, limiting coverage to cases with an analytic mechanism-to-law mapping.

    Source: §6 · p. 9

Abstract

from arXiv · show

Scientific discovery requires not only recovering mathematical laws that describe observable behavior, but also identifying the mechanisms that generate them. Existing benchmarks for symbolic regression and scientific agents primarily evaluate phenomenal-law recovery, leaving mechanism discovery largely untested. We introduce MechBench, a benchmark that explicitly separates these two capabilities. Each task is defined by a mechanistic model, a structured set of scientifically meaningful relations whose joint consequences entail an observable phenomenal law, while agents receive only observational data and scientific context. We evaluate mechanism recovery through mechanism probes, which query internal scientific consequences that cannot be inferred from the phenomenal law alone. To reduce reliance on memorized textbook mechanisms, we construct unfamiliar variants through controlled, scientifically interpretable mutations of canonical mechanisms, and screen for mechanistic indistinguishability to exclude ambiguous instances admitting comparable competing mechanisms. Experiments across representative scientific agents reveal a substantial phenomenal--mechanism recovery gap: for Codex with GPT-5.6-sol, phenomenal-law accuracy reaches 35.00% on the Core-set while mechanism accuracy is only 13.75%, with mechanism recovery failing in 64.29% of cases where the phenomenal law is correctly recovered. The gap widens as mechanisms become increasingly mutated, and even providing the correct phenomenal law leaves mechanism recovery below 50%. These results reveal a substantial generalization gap in mechanistic reasoning and establish mechanism discovery as a distinct challenge beyond recovering observable scientific laws.

1 Introduction

Scientific discovery requires distinguishing laws that describe observable behavior from mechanisms that generate it. MechBench tests this distinction, and experiments find mechanism recovery substantially weaker than phenomenal-law recovery.

1 Introduction

  • Phenomenal models characterize observable relationships, whereas mechanistic models specify the internal causal structure generating them.Mechanistic models make commitments about internal quantities that the observable law does not specify.Source: §1 · pp. 1–2
  • Existing scientific-agent benchmarks chiefly test recovery of relations reproducing observable behavior, not reconstruction of the mechanisms underlying those relations.Mechanism evaluation must also distinguish reconstruction from textbook recall and address competing mechanisms with the same observable law.Source: §1 · p. 2
  • MechBench separates observable-law recovery from reconstruction of the structured mechanisms whose relations jointly entail those laws.Mechanism probes query internal consequences not inferable from the phenomenal law and scientific context alone.Source: §1 · p. 2
  • 35.00% phenomenal-law accuracy versus 13.75% mechanism accuracy for Codex with GPT-5.6-sol on the Core-set illustrates the recovery gap.Mechanism recovery also fails in 64.29% of cases where the phenomenal law is correctly recovered.Source: §1 · p. 2

2 Related Work

Prior scientific equation-discovery systems and benchmarks increasingly use tools, unfamiliar tasks, and scientific context, but primarily evaluate observable-law recovery. MechBench instead evaluates recovery of the internal scientific relations underlying those laws.

2 Related Work

  • Scientific-agent systems extend symbolic regression with prior knowledge, program synthesis, data analysis, tools, and iterative hypothesis refinement, while still targeting observable relations.Examples include LLM-SR, SR-Scientist, KeplerAgent, and A-SR.Source: §2 · p. 2
  • Classical symbolic-regression benchmarks such as Feynman and SRBench evaluate recovery of compact equations from predefined observations.The supplied passage also names Udrescu & Tegmark (2020), La Cava et al. (2021), and Matsubara et al. (2022).Source: §2 · pp. 2–3
  • Recent law-discovery benchmarks increase task unfamiliarity, interactivity, and scientific grounding, but their primary evaluation target remains the recovered observable law.MechBench evaluates structured mechanistic models and their internal consequences using controlled variants, probes, and indistinguishability screening.Source: §2 · p. 3

3 From Phenomenal Laws to Mechanistic Discovery

MechBench formalizes a mechanism as internal scientific relations that entail an observable law, then evaluates additional consequences that distinguish mechanisms beyond that law. It screens out instances where comparably admissible competing mechanisms make recovery ambiguous.

3.1 Formalizing Mechanistic Discovery

  • A mechanism comprises scientifically meaningful relations involving observable variables and unobservable internal quantities; eliminating the internal quantities yields the phenomenal law.The mechanism therefore contains structure beyond the resulting observable relationship.Source: §3.1 · p. 3
  • Agents reconstruct mechanisms from observational data and scientific context without access to the reference mechanism or its internal quantities.The passage describes observations sampled from the phenomenal law induced by the reference mechanism.Source: §3.1 · pp. 3–4

3.2 Mechanism Probes

  • Mechanism probes test internal consequences because equivalent mechanism representations need not match literally, and distinct mechanisms can entail the same phenomenal law.Probes are selected to capture reference-mechanism consequences that cannot be inferred from the phenomenal law and context alone; their expressions are objectively checkable.Source: §3.2 · p. 4

3.3 Mechanistic Indistinguishability

  • A competing mechanism that is comparably admissible under the scientific context and entails the same phenomenal law makes recovery of the designated mechanism ambiguous.Probe agreement does not establish which such competitor is more scientifically plausible.Source: §3.3 · p. 4
  • Figure 2 depicts benchmark construction and evaluation, including mutation of canonical mechanisms, data and held-out tests, competitor screening, and two recovery signals.The two signals are phenomenal-law recovery and mechanism probes.Source: Figure 2 · p. 5

4 Benchmark Construction

MechBench builds unfamiliar mechanisms from scientifically grounded mutations, derives observations and held-out tests from their phenomenal laws, and screens for competing mechanisms. Submissions are evaluated both for observable-law recovery and for mechanistic consequences.

4.1 Constructing Mechanism Variants

  • Canonical mechanisms from several sciences serve as blueprints for unfamiliar variants, addressing the risk that agents reproduce mechanisms already encountered in pretraining.Mutations favor scientifically interpretable local changes over arbitrary symbolic perturbations.Source: §4.1 · p. 5
  • Compatible mutations can be composed, while task families retain a shared scientific scaffold and vary which and how many canonical elements are modified.For each mechanism, the phenomenal law is derived by eliminating internal quantities and used to generate observations; input ranges are split into ID interpolation and OOD extrapolation regions.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 Constructing Benchmark Instances

  • Candidate alternatives that reproduce the same phenomenal law are grounds to revise or exclude a variant when they require comparable or lower complexity and no greater departure from scientific background knowledge.The construction procedure searches for alternatives that modify other relations than the intended mutation.Source: §4.2 · p. 6

4.3 Benchmark Evaluation Protocol

  • Evaluation derives a phenomenal law from each submitted mechanism and compares it with the reference using symbolic and numerical equivalence.Mechanism recovery separately queries probes using the submitted mechanism and compares its predicted relations with reference relations; probes are queried independently.Source: §4.3 · p. 6

5 Experiments

The experiments compare phenomenal-law and mechanism recovery across scientific agents, datasets, mutation levels, and task families. Mechanism recovery consistently lags law recovery, including when the correct law is provided.

5.1 Experimental Setup

  • The Core-set contains 80 instances from eight task families; the Full-set contains 512 instances from 16 families, spanning four scientific fields.Seven agent configurations are evaluated on the Core-set, while Codex with GPT-5.6-sol and GLM-5.3-flash are also evaluated on the Full-set; Gold-P supplies the reference phenomenal law directly.Source: §5.1 · p. 6

5.2 The Phenomenal–Mechanism Recovery Gap

  • Across all seven agent configurations, mechanism accuracy is lower than phenomenal-law accuracy; Codex with GPT-5.6-sol scores 35.00% versus 13.75% on the Core-set.On the Full-set, the corresponding scores are 16.02% and 7.81%.Source: §5.2 · p. 7, Table 1 · p. 7
  • For Codex with GPT-5.6-sol, conditional mechanism failure is 64.29% on the Core-set and 57.32% on the Full-set among cases with correctly recovered laws.Across other configurations, Core-set conditional failure rates range from 49.07% to 67.22%.Source: §5.2 · p. 7
  • As mutations accumulate, mechanism accuracy deteriorates more sharply than phenomenal-law accuracy, and conditional mechanism failure reaches 100% at three Core-set and four Full-set mutations.Original-instance conditional failure is approximately 33% on the Core-set and 17% on the Full-set.Source: §5.2 · p. 7, Figure 3 · p. 8

5.3 Oracle Control: Knowing the Law Is Not Enough

  • With Gold-P Direct-Ask, GPT-5.6-sol mechanism accuracy reaches 45.00% on the Core-set and 49.02% on the Full-set, yet remains below 50% on both.Across both datasets, every evaluated model has higher mechanism accuracy under Gold-P Direct-Ask than standard discovery.Source: §5.3 · p. 7, Table 2 · p. 8
  • Under Gold-P, GPT-5.6-sol mechanism accuracy falls from 87.50% on Original instances to 12.50% on the Core-set and 22.50% on the Full-set at four mutations.The two flash models remain near zero across most multi-mutation groups; the results indicate a generalization gap from familiar to novel mechanisms.Source: §5.3 · pp. 7–8, Figure 4 · p. 8

5.4 Comparisons Across Models, Harnesses, and Task Families

  • With Codex fixed, GPT-5.6-sol outperforms the two flash models, while evaluated harnesses and scientific domains show no detected significant differences; task-family recovery profiles vary substantially.Inclined Rolling has relatively high law recovery but low mechanism recovery, while Competitive Inhibition Pulse Reaction shows the opposite pattern.Source: §5.4 · pp. 8–9, Table 3 · p. 8

6 Conclusion

MechBench evaluates mechanism discovery separately from phenomenal-law recovery using controlled mutations, mechanism probes, and screening for mechanistic indistinguishability. The experiments find a gap that widens for mutated mechanisms, while the benchmark remains limited to cases admitting tractable symbolic elimination.

6 Conclusion

  • MechBench combines controlled mechanism mutations, mechanism probes, and indistinguishability screening to evaluate internal scientific structure beyond observational agreement.Across representative agents, the mechanism-recovery gap widens as mechanisms depart from canonical forms, even when the correct phenomenal law is provided.Source: §6 · p. 9
  • The benchmark currently covers mechanisms whose observable phenomenal laws can be obtained through tractable symbolic elimination.Systems requiring nonlinear or transcendental solutions, differential-equation integration, or numerical simulation are outside this scope.Source: §6 · p. 9

AI use statement

  • Generative AI tools assisted with ideation, benchmark and experiment design, synthetic instance construction, coding, literature work, and presentation; the authors report reviewing AI-assisted work.The authors report manually checking generated ideas for plagiarism and verifying and testing generated code.Source: AI use statement · p. 9

Ethics statement

  • The work uses synthetic observations from scientific mechanism models and reports no human subjects, personal or private data, or decisions affecting individuals.The authors foresee no ethical concerns.Source: Ethics statement · p. 9

Reproducibility statement

  • The authors state that benchmark documentation, code, task instances, and data are publicly available.The reproducibility statement gives the MechBench GitHub repository.Source: Reproducibility statement · p. 9

A.1 Task Families and Reference Mechanisms

MechBench covers 16 task families across four scientific domains, then constructs mutation variants and evaluates their mechanisms through internal-quantity probes. Its Full-set and Core-set organize these variants, while composition rules preserve compatible mechanism structures.

A.1 Task Families and Reference Mechanisms

  • The 16 task families span physics, chemistry, biology, and materials science, with each Original mechanism grounded in a standard scientific construction.Examples include inverse-square orbital dynamics, enzyme-state balances, promoter occupancy, and layered heat conduction.Source: §A.1 · pp. 11–12, Table 4 · p. 12

A.2 Full-set and Core-set Membership

  • The Full-set contains 512 instances across all 16 families, while the mutation-stratified Core-set contains 80 instances from eight selected families.Each Full-set family includes its Original and all 31 nonempty combinations of five base mutations.Source: §A.2 · p. 13

A.3 Mechanism Mutations and Their Composition

  • Mutations modify mechanism relations before the phenomenal law is derived, and compatible combinations merge shared states or balances without duplicating contributions.Retained compositions must yield unique explicit solutions for internal quantities and the target; variants are constructed before internal-variable elimination.Source: §A.3 · pp. 13, 16

A.4 Mechanism Probe Construction

  • Mechanism probes query scientifically meaningful internal quantities that contribute to the phenomenal law but are not determined by that law and context alone.Comparable plausible mechanisms must agree on a probe quantity; probe sets adapt to active relations, with one to five probes across Full-set instances.Source: §A.4 · pp. 16–17

A.5 Observation Generation and Task Specification

The benchmark generates observations from solved reference mechanisms, exposing scientific context and training data while withholding variant-specific structure and evaluation targets. It compares standard discovery with symbolic-regression and direct-ask controls.

A.5 Observation Generation and Task Specification

  • Training and held-out ID samples use the same input region, while OOD samples jointly extrapolate all source variables beyond their training boundaries.Sampling ranges are selected for each instance according to scientific meaning, characteristic scale, and valid operating regime.Source: §A.5 · p. 17
  • During discovery, agents receive task and variable descriptions plus training observations, but not active mutations, internal variables, reference laws, probes, or held-out sets.A feedback script evaluates submitted equations against training observations without revealing reference mechanisms or other hidden evaluation information.Source: §A.5 · pp. 17–18

B.1 Evaluated Configurations

  • All seven standard-discovery configurations are evaluated on the Core-set, while Full-set standard discovery covers Codex with GPT-5.6-sol and GLM-5.3-flash.PySR, PySR + Direct-Ask, and Gold-P Direct-Ask are evaluated on both sets.Source: §B.1 · p. 18

B.2 Standard Discovery and Baselines

  • PySR serves as a non-LLM baseline for recovering phenomenal laws, and PySR + Direct-Ask tests probe answering after that equation is recovered.The latter provides the recovered equation to DeepSeek-v4-flash-0731 for independent mechanism-probe answers.Source: §B.2 · p. 19
  • Gold-P Direct-Ask supplies the reference phenomenal law but withholds the reference mechanism and probe expressions, testing whether internal structure can be reconstructed from the law and context.Each probe is asked independently in a tool-free query.Source: §B.2 · p. 19

B.3 Evaluation Metrics

The evaluation distinguishes exact symbolic recovery of phenomenal laws and mechanisms from held-out numerical agreement. It also specifies mutation-based reporting subsets and the implementation conditions for standard and direct-ask evaluations.

B.3 Evaluation Metrics

  • A task counts as mechanistically correct only when every probe is symbolically correct, and this score is unconditional on phenomenal-law recovery.Held-out numerical accuracy instead counts a task correct only when its maximum relative error across the split is at most 10%.Source: §B.3 · p. 19

B.4 Mutation Strata and Reported Subsets

  • The Full-set includes Originals and all mutation combinations, while variant-only analyses exclude Originals, leaving 496 Full-set and 72 Core-set variants.Results are also stratified by mutation count, with Core-set membership fixed and mutation-stratified.Source: §B.4 · pp. 19–20

B.5 Implementation and Runtime Settings

  • Standard discovery is limited to 900 seconds per task for Codex, Claude Code, DeepSeek Harness, and PySR, with independent probe turns capped at 120 seconds.The harness prompts share the objective, data, feedback access, submission contract, time budget, and isolation rules, while retaining harness-specific wording and native instructions.Source: §B.5 · p. 20

B.6 Direct-Ask Prompt and Runtime

  • Gold-P Direct-Ask substitutes the exact reference phenomenal equation and relevant numerical constants into the common probe template, without providing the defining mechanism equations.Probe queries are fresh, one-turn, tool-free requests.Source: §B.6 · p. 21

C.1 Complete Recovery and Numerical-Accuracy Results

Across domains and task families, phenomenal-law recovery does not reliably imply complete mechanism recovery. Held-out numerical agreement is higher than exact symbolic recovery in the reported Core-set results.

C.1 Complete Recovery and Numerical-Accuracy Results

  • SA(M) is a strict task-level measure requiring every mechanism probe for an instance to be symbolically correct.The complete domain-level symbolic-recovery tables underlie the aggregate values in Table 1.Source: §C.1 · p. 22
  • 40.00% phenomenal-law recovery versus 10.00% mechanism recovery in biology, and 20.00% versus 0.00% in materials, illustrate Codex GPT-5.6-sol's domain-level gap.The configuration has the highest overall phenomenal-law and mechanism recovery on both sets and leads mechanism recovery in every Full-set domain.Source: §C.1 · p. 22
  • 50.00% phenomenal recovery but 6.25% mechanism recovery for Inclined Rolling contrasts with 31.25% and 25.00%, respectively, for Competitive Inhibition Pulse Reaction.Across Full-set task families, greater phenomenal recoverability does not necessarily coincide with greater mechanism recoverability.Source: §C.1 · p. 22
  • 55.00% AccID_0.1 and 43.75% AccOOD_0.1 compare with 35.00% SA(P) for Codex GPT-5.6-sol on the Core-set.Numerical agreement is less stringent than exact symbolic recovery, and OOD accuracy is lower than ID accuracy for every reported agent configuration.Source: §C.1 · p. 22

C.2 Results by Mutation Count and Variant Subset

Recovery declines as mechanisms accumulate mutations, with strict mechanism recovery falling faster; excluding Original instances also lowers Codex recovery, while smaller configurations rarely recover heavily mutated mechanisms.

Variant-only Results

  • Excluding Original instances reduced Codex with GPT-5.6-sol SA(P) from 35.00% to 30.56% on the Core-set and from 16.02% to 14.11% on the Full-set.The variant-only subsets contain 72 Core-set and 496 Full-set instances.Source: §C.2 · p. 22
  • After excluding Originals, Codex with GPT-5.6-sol strict SA(M) fell from 13.75% to 9.72% on the Core-set and from 7.81% to 6.05% on the Full-set.The variant-only subsets contain 72 Core-set and 496 Full-set instances.Source: §C.2 · p. 22
  • Other standard discovery configurations reached at most 4.17% phenomenal and 1.39% mechanism recovery on Core-set variants; GLM-5.3-flash reached 3.23% and 0.81% on Full-set variants.For smaller models, successful recoveries were concentrated among Original and single-mutation instances; none recovered a complete Core-set mechanism at three or more mutations.Source: §C.2 · p. 22

Results by Mutation Count

  • On the Core-set, strict SA(M) fell from 50.00% on Original instances to 37.50% with one mutation, 6.25% with two, and zero with three or more.The Full-set fell from 62.50% on Original instances to zero by four mutations.Source: §C.2 · p. 22
  • Core-set SA(P) for Codex with GPT-5.6-sol fell from 75.00% on Original instances to 6.25% with four mutations and 0.00% with five.Full-set phenomenal recovery also fell from 75.00% on Original instances to zero at five mutations.Source: §C.2 · p. 22

C.3 Statistical Comparisons

GPT-5.6-sol significantly outperformed both flash models across the three evaluated outcomes, while comparisons found no significant differences between flash models, evaluated harnesses, or scientific domains.

Base-model comparisons

  • GPT-5.6-sol exceeded GLM-5.3-flash by 28.75 percentage points in SA(P), 10.83 in SA(M), and 22.92 in OOD Acc0.1, with adjusted p < .001, p = .008, and p < .001.The comparisons use paired tests over 80 tasks with Holm correction across six model–metric comparisons.Source: §C.3 · p. 23, Table 18 · p. 25
  • GPT-5.6-sol exceeded DeepSeek-v4-flash-0731 by 27.08 points in SA(P), 10.00 in SA(M), and 21.67 in OOD Acc0.1, with the same adjusted significance levels.The adjusted p values were < .001, .008, and < .001, respectively.Source: §C.3 · p. 23, Table 18 · p. 25

Base-model and harness comparisons

  • Comparisons detected no significant differences between the two flash models or among evaluated harnesses across SA(P), SA(M), and OOD Acc0.1.Flash-model paired-test p values were .234, .414, and .662; harness omnibus-test p values were .867, .607, and .264.Source: §C.3 · pp. 23–24, Table 18 · p. 25

Domain comparisons

  • No statistically significant differences appeared among physics, chemistry, biology, and materials science for SA(P) (p = .363), strict SA(M) (p = .092), or OOD Acc0.1 (p = .149).Physics had the highest descriptive mean on all three metrics, but the evidence did not support a general domain-level ordering.Source: §C.3 · p. 24, Table 18 · p. 25
Loading 2609.35515v1…