PaperlayerView exact revision on arXiv

Source-linked AI summary

MechReasoner: A Simulator and Benchmark for Mechanistic Reasoning in Qualitative Physics

Danilo Gusicuma, André Freitas

arXiv:2609.34636v1cs.AI

TL;DR

LLMs can produce fluent mechanistic descriptions without reliably predicting phenomena, while existing evaluations often fail to test whether claims follow from mechanism constraints. MechReasoner uses an executable qualitative reference and benchmark to test this relation; GPT-5.5 accuracy declines from 76.1% in B1 to 38.0% in B4 as mechanistic complexity rises.

  • Problem

    Existing benchmarks rarely test whether a claim is licensed by a mechanism, including whether transitions are possible, alternatives are omitted, or evidence follows from stated laws.

    Source: §1 · p. 1

  • Method

    MechReasoner uses a qualitative simulator to materialize a constrained possibility space as admissible interpretations, an episode graph, and auditable evidence for six task families.

    Source: §1 · pp. 1–2

  • Results

    Every evaluated model scored lower in B4 than B1; GPT-5.5 fell from 76.1% to 38.0%, and length controls left the complexity association intact.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The benchmark makes mechanistic claims testable against admissible interpretations or reachable episode paths while retaining underdetermination and quantified judgments.

    Source: §5 · p. 7

    The benchmark covers 18 mechanisms and six task families, and results for agentic-ready models characterize direct inference rather than agent loops.

    Source: §6 · p. 7

Abstract

from arXiv · show

This work introduces MechReasoner, a mechanistic qualitative simulator grounded in confluence-based qualitative physics, together with a benchmark for mechanistic inference. Current large language models (LLMs) generate fluent mechanistic descriptions that do not reliably follow from underlying structural and causal constraints. The benchmark tests whether answers preserve simulator-licensed ambiguity, quantified claims, episode-graph transition evidence, repairs, and trace-support judgments. Its 1,120 items are generated deterministically from admissible interpretation sets, component states, scenario restrictions, confluence constraints, and derivation steps across 18 catalog mechanisms and six task families. Each mechanism undergoes converter checks of structure and topology and behavioral checks against quantitative simulations. GPT-5.5 accuracy decreases as family-specific mechanistic complexity increases, from 76.1% in the lowest-complexity bucket (B1) to 38.0% in the highest-complexity bucket (B4). The negative association remains after controls for rendered-prompt and expected-answer length. These results show that qualitative simulators can support auditable NLP benchmarks for mechanistic inference.

1 Introduction

MechReasoner addresses the gap between fluent mechanistic descriptions and claims that actually follow from a mechanism’s structural and causal constraints. It uses an executable qualitative reference to evaluate such claims and reports lower model accuracy at higher mechanistic complexity.

1 Introduction

  • Qualitative physics represents structural and causal constraints without exact numerical parameters and preserves alternative behaviors when information is incomplete.The simulator enumerates admissible interpretations, reachable episode paths, and the evidence licensing them.Source: §1 · p. 1
  • Existing benchmarks may accept correct labels despite impossible transitions, omitted alternatives, or evidence unsupported by stated laws.The paper argues that evaluation needs a semantic reference enumerating admissible states and histories while preserving transition and derivation evidence.Source: §1 · p. 1
  • Six task families query a shared constrained possibility space for state membership, path existence, necessary effects, reachable changes, repairs, and derivation support.The simulator materializes that space as admissible interpretations, an episode graph, and auditable evidence.Source: §1 · p. 1
  • Every evaluated endpoint scored lower in B4 than B1; GPT-5.5 declined from 76.1% to 38.0%.The endpoints answered the same 1,120 prompts.Source: §1 · p. 2
  • The release contains 1,120 items across 18 mechanisms and six task families, with inspectable gold evidence and converter and quantitative checks across 341 observed scenarios.It also includes the typed representation, simulator, deterministic generators, manifests, evaluators, and analysis artifacts.Source: §1 · p. 2

2 Mechanistic Abstraction and Benchmark

The benchmark formalizes mechanisms as constrained qualitative possibility spaces, solves their interpretations, and constructs episode graphs with licensed transitions. It uses these shared semantic objects to generate and score deterministic tasks, validate the mechanisms, and evaluate model responses.

2.1 Formal Problem Statement

  • For each state and scenario, active constraints yield finite admissible interpretations; graph nodes pair states with interpretations, while transitions require licensing declarations and causal, continuity, and endpoint checks.At the interpretation level, a claim is plausible if some admissible interpretation supports it and necessary if all do.Source: §2.1 · p. 3

2.2 Simulator Architecture

  • The simulator records solved interpretations, forced facts, and evidence, then builds episode-graph edges; deterministic generation applies family-specific queries and complexity scores before validation and selection.The resulting tasks, gold solutions, manifests, and completion hashes are persisted.Source: §2.2 · p. 3

Quantitative–qualitative consistency check

  • A consistency check compared qualitative and numerical results across 341 observed scenarios from all 18 mechanisms, finding zero contradictions and zero inconclusive cases.The numerical and qualitative results followed separate execution paths and did not use benchmark tasks or gold solutions.Source: Quantitative–qualitative consistency… · p. 3

2.3 Benchmark as a Mechanistic Abstraction

  • The benchmark treats each mechanism’s interpretation sets, episode graph, and rule-evidence records as a semantic abstraction, then tests whether reasoners recover queried properties from its serialized representation.This separates mechanistic reasoning from imitation of a particular simulator or predicted trajectory.Source: §2.3 · p. 3

Task semantics

  • The six task families test distinct semantics, including state consistency, witnessed episode paths, necessity, reachable changes, universal repair status, and trace-step support.Table 1’s illustrative figures use green for valid or forced elements, red for invalid, blocked, or unsupported ones, and neutral circles for candidates.Source: Task semantics · p. 4, Table 1 · p. 4

Benchmark governance

  • The audited comparator pool is deduplicated by normalized prompt, and completion markers bind the catalog, generator, manifest, solutions, and task files.All prompts fit the stated 80,000-token ceiling; the maximum is 73,866 tokens.Source: Benchmark governance · p. 4

Evaluator protocol

  • Models receive the same persisted prompts without tools or repository access; the scorer normalizes responses by family and compares them exactly with gold.End-to-end accuracy retains every model–item record, while family-specific measures include transition-set F1 and trace count-field accuracy; complexity metrics are computed without expected answers.Source: Evaluator protocol · p. 4, Table 2 · p. 5

3 Empirical Analysis

Across the common benchmark, model accuracy falls with increasing mechanistic complexity, while performance varies substantially by task family. The section also defines the benchmark’s model, complexity-bin, and dataset setup.

Model Selection Criteria

  • The nine evaluated endpoints span general and reasoning models, with and without documented agentic readiness, but all are tested by direct LLM inference without tools or action loops.GPT-5.5 supplies the frontier reference point.Source: Model Selection Criteria · p. 5

Mechanistic Complexity

  • Complexity is measured by a deterministic, answer-independent simulator-derived scalar for each task family, with retained items assigned to tie-preserving near-quartile bins.B1–B4 denote within-family strata, not comparable absolute complexity values across families; the retained-item bins contain 309/273/267/271 items.Source: Mechanistic Complexity · p. 5

Dataset Characterization

  • Deduplication left 1,120 unique questions from 1,141 generated task-ID records, across six task families and 18 mechanisms; the 608-question shortfall was not padded.The 108 family–mechanism cells targeted 1,728 generation slots.Source: Dataset Characterization · p. 5

Frontier and reasoning models lead the full benchmark

  • Across task families, GPT-5.5 accuracy is highest on state consistency (79.7%), followed by episode-graph transitions (78.0%) and trace faithfulness (75.4%), and lowest on necessity (23.1%).Component scores are 98.4% candidate–case accuracy for state consistency, 89.2% plausibility character accuracy, 65.6% necessity character accuracy, 88.3% transition-set F1, 87.3% functional-recovery character accuracy, and 75.4% trace count-field accuracy.Source: Frontier and reasoning models lead the… · pp. 5–6

3.2 Results

  • Across nine models, pooled accuracy falls from 23.7% in B1 to 8.7% in B4; every model scores lower in B4, with GPT-5.5 declining in all six families.GPT-5.5’s family-specific declines range from 19.6 to 75.0 points; the adjusted per-bin odds ratio is 0.619, with a mechanism-cluster CR1–t17 95% interval of [0.547, 0.701]. The association is within-family and not causal.Source: §3.2 · p. 5, Figure 2 · p. 6

Frontier and reasoning models lead the full benchmark

4 Related Work

MechReasoner builds on qualitative physics but uses its simulator to construct and score a benchmark grounded in executable mechanisms. It differs from text-based, single-successor, and broader physics question-answering evaluations by testing claims against constrained states and histories.

Classical qualitative reasoning and simulation

  • MechReasoner retains the full interpretation set to support quantified claims, constrained repair analysis, and explanation checks, building on qualitative-physics approaches to behavior-from-structure reasoning.Its objective is benchmark construction and scoring using released qualitative semantics, rather than qualitative analysis of engineering models during design.Source: §4 · pp. 6–7

LLM-era simulator grounding and evaluation

  • Unlike Mind’s Eye, which supplies simulated outcomes as hints, MechReasoner uses its simulator to construct and score tasks while models receive no simulator outcomes.The gold preserves the ambiguity set of admissible interpretations, allowing possible and necessary conclusions to be distinguished under underdetermination.Source: §4 · p. 7
  • Unlike LLM-Sim’s prediction of one subsequent game state or state difference, MechReasoner evaluates quantified queries over documented admissible states and histories.LLM-Sim separately measures action-driven and environment-driven transitions.Source: §4 · p. 7

Process and physical commonsense benchmarks from language; Recent physics benchmarks

  • Text-based commonsense and newer physics benchmarks offer linguistic coverage or executable artifacts, whereas MechReasoner evaluates against executable mechanism models and their state and transition semantics.The cited resources include ProPara, WIQA, PIQA, NEWTON, PhysReason, and SymPyBench.Source: §4 · p. 7

5 Conclusion

MechReasoner makes mechanistic claims testable against a qualitative-physics possibility space, preserving ambiguity while measuring model performance. Its results show a complexity association that persists under prompt- and answer-length controls.

5 Conclusion

  • MechReasoner tests claims against a constrained possibility space, with conclusions licensed by admissible interpretations or reachable episode paths.Its queries preserve underdetermination and support quantified judgments over the same encoded mechanism.Source: §5 · p. 7
  • The prompt-text symbolic baseline solves every item, while direct-inference accuracy declines with complexity and the association remains after prompt- and answer-length controls.Higher component scores than exact scores indicate that partial structural recovery may fail to yield a coherent final judgment.Source: §5 · p. 7

6 Limitations

The benchmark is bounded in scope and validation, and its reported complexity association retains a surface-form confound despite length controls. Validation checks and uncertainty estimates also have specific limits.

6 Limitations

  • The benchmark covers 18 mechanisms and six task families, not agentic deployment, interactive tools, multimodal perception, continuous prediction, open-ended explanation, or safe engineering practice.Results for agentic-ready models characterize direct inference, and benchmark success does not establish mechanistic internal computation.Source: §6 · p. 7
  • Prompt- and answer-length controls leave the complexity association negative and do not remove the surface-form confound.Exact accuracy also compounds errors across fields.Source: §6 · p. 7
  • Generation coverage is nonuniform, and reported intervals condition on one persisted response per item, excluding variability across repeated model inferences.The intervals capture variation from resampling the mechanism catalog; protocol failures remain in the end-to-end denominator.Source: §6 · p. 7
  • Quantitative consistency checks cover 341 observed scenarios but do not fully validate benchmark simulation or existential claims lacking observed supporting behavior.The prompt-text symbolic baseline shares the released semantics implementation, so its 100% accuracy establishes prompt completeness and implementation consistency, not independent gold correctness.Source: §6 · p. 7

Ethical Statement

The tasks are synthetic and use no human-subject data; their scores do not establish real-world engineering competence or certify safety-critical deployment.

Ethical Statement

  • Tasks are synthetic and contain no human-subject data.Source: Ethical Statement · p. 7
  • Scores do not establish real-world engineering competence or certify safety-critical deployment.Source: Ethical Statement · p. 7
  • AI assistance supported wording review, while the authors independently verified all claims and remain responsible for the submission.Source: Ethical Statement · p. 7

A.1 Simulator Architecture Diagram

Figure 4 summarizes the artifact and trust boundaries from bounded Modelica conversion through benchmark-item persistence.

A.1 Simulator Architecture Diagram

  • Figure 4 traces the artifact and trust boundaries from bounded Modelica conversion through benchmark-item persistence.Source: §A.1 · p. 8

A.2 Accuracy Surface Controls

The exploratory robustness analysis stacks nine models over the common 1,120-question benchmark and uses conditional logistic regressions stratified by model, task family, and mechanism. Its adjusted fit adds prompt and expected-answer lengths, while the authors characterize the post-hoc estimate as associational sensitivity evidence rather than a causal or confirmatory result.

A.2 Accuracy Surface Controls

  • The exploratory analysis stacks nine models over 1,120 questions, retaining protocol failures as incorrect and using B1/B2/B3/B4 counts of 309/273/267/271 questions per model.This yields 10,080 model–item records across six task families.Source: §A.2 · p. 8
  • Conditional logistic regressions stratify by model, task family, and mechanism; the adjusted fit adds rendered-prompt and expected-answer character counts.Source: §A.2 · p. 8
  • The fits use 342 informative strata and 4,841 records, with all 18 mechanisms retained as clusters; 423 invariant strata contribute no conditional likelihood.Intervals use mechanism-cluster CR1 covariance with a t17 reference distribution, and ordinal p-values use a fully enumerated wild-cluster score test.Source: §A.2 · p. 8
  • Both ordinal fits converged, and the adjusted fit had a full-rank information matrix with condition number 117.0.Its final score infinity norm was 4.63 × 10−13; the largest absolute coefficient was 3.73.Source: §A.2 · p. 8
  • The post-hoc estimate is associational sensitivity evidence, not a causal or confirmatory claim, given nonuniform coverage, only 18 cluster contributions, and unrecorded prompt-content differences.The shared ordinal coefficient does not establish equal gradients across the nine models.Source: §A.2 · p. 9

A.3 Outcome-Separated Metrics and Baselines

The evaluation separates infrastructure, protocol, semantic, and scoring outcomes so end-to-end accuracy reflects every model–item record. It pairs these measures with a diagnostic baseline and reports model-level results and uncertainty.

A.3 Outcome-Separated Metrics and Baselines

  • Compliance measures acceptance by the normalizer, distinguishing valid payloads from empty, rejected, or otherwise invalid responses.Infrastructure failures are excluded from the accepted-payload denominators.Source: §A.3 · p. 9
  • End-to-end accuracy retains all model–item records, while semantic scores condition on normalized payloads and component measures capture family-specific outcomes.Infrastructure and protocol failures receive zero in the end-to-end component-score mean; the conditional semantic mean averages accepted payloads.Source: §A.3 · p. 9
  • Accuracy intervals use 10,000 deterministically seeded, mechanism-level cluster-bootstrap replicates, with shared draws preserving paired model comparisons.Resampling entire item clusters by mechanism preserves within-mechanism dependence and item-weighted accuracy.Source: §A.3 · p. 9
  • The majority/constant baseline is a diagnostic floor fitted to released gold-label marginals, whereas the prompt-text symbolic parser solves all 1,120 items with 100% accuracy.The parser reconstructs tasks from rendered prompts and uses the public qualitative solver without reading solutions, catalog files, generator objects, or simulator caches.Source: §A.3 · p. 9
  • 1,296 protocol failures are reported across the complete per-model, per-family evaluation, alongside end-to-end accuracy for every model and family.Ten Qwen3-30B requests returned empty responses and are classified as protocol failures.Source: §A.3 · p. 9

A.4 Operational Semantic Definitions

This appendix specifies the simulator’s operational semantics and task-specific rules, including deterministic task construction, answer extraction and normalization, and evaluation and prompt protocols. It also documents bounded numerical checks against the qualitative simulator.

A.4 Operational Semantic Definitions

  • For each composite state and scenario, the solver restricts variable domains, activates local and connection laws, and returns finite assignments satisfying every confluence.A confluence is not satisfied by a zero term combined only with terms of one sign.Source: §A.4 · p. 9
  • Episode edges are admitted from causes only when endpoint checks enforce event ordering, causal support, continuity, derivative-consistent changes, and declared component transitions.Boundary causes use nearest directed termination events; intrastate causes are available only when mandatory point-cell departures do not preempt them.Source: §A.4 · p. 10
  • Task labels use exact bounded paths for plausibility, reachable edges for necessity, witnessed transitions, repair safety and deadlines, and counts of unfaithful trace steps.State consistency checks whether a candidate composite state has a satisfying interpretation under the scenario’s restrictions and active constraints.Source: §A.4 · p. 11

A.5 Candidate Construction, Validation, and Rendering

  • All six generators build deterministic candidate pools from validated mechanisms, compute gold answers with exact solvers before rendering, and apply frozen family-specific checks and selection rules.Selection targets four tasks per supported family–mechanism–level cell; cells remain partial or unsupported when search or prompt limits prevent filling them.Source: §A.5 · p. 11, Table 7 · p. 12

A.6 Scenario-Level Simulator Protocol

  • The simulator separates intrastate solving from episode-graph construction, using factorized satisfying relations and exact propagation and decision-diagram methods to support bounded reachability.The simulation session separately applies declaration, guard, cause, continuity, and endpoint checks to construct the reachable graph.Source: §A.6 · p. 12

A.7 Prompt and Evaluator Contract

  • Evaluation sends the persisted task content in a shared two-message envelope, with no task-family schema out of band and only generic JSON mode where supported.Each backend receives one request; model-directed tools, repository access, file inspection, and command execution are outside the comparison protocol.Source: §A.7 · p. 12

Request Controls

  • The direct-inference protocol sets concurrency to 20, a 600-second timeout, at least 80,000 tokens of input capacity, and a 20,000-token output ceiling subject to profile limits.Temperature zero and low reasoning effort are used only for profiles that support those controls.Source: Request Controls · p. 12

Current Rendered Prompt Excerpts

  • Rendered excerpts illustrate task prompts that supply mechanism evidence and explicit semantics, including exact episode-graph rules for plausibility and necessity judgments.For plausibility, all checkpoint observations must be witnessed by the same path of exactly the supplied horizon.Source: Current Rendered Prompt Excerpts · pp. 13–14
  • For episode-graph transitions, return distinct component-state changes witnessed by admissible edges before the horizon, including edges into but not out of the horizon.Targets must satisfy declared state-pair and event constraints plus consistency, causality, continuity, and the qualitative mean-value rule; interpretation-only edges are excluded. Functional recovery evaluates each repair policy universally over its named outcomes and prioritizes unsafe states, then deadline goal failure, then recovery.Source: Current Rendered Prompt Excerpts · p. 15
  • Trace-faithfulness cases restart from initial domains, require each before-set to match the current domain, and judge cited laws by their exact supported target-cell sets.Every displayed after-set is applied even after an unfaithful citation, and cases are independent.Source: Current Rendered Prompt Excerpts · pp. 15–16

Answer extraction and normalization rules

  • Extraction parses the complete response first, then a matching fenced object if needed, and only then the span from the first opening brace to the last closing brace.If the parsed value contains an answer field, that field becomes the payload; otherwise the parsed value itself is normalized. Family normalizers discard unrelated top-level fields and do not use expected answers to repair or infer model values.Source: Answer extraction and normalization… · p. 16
  • Family normalizers enforce documented payload shapes, widths, labels, and types; retries do not combine samples or repair invalid model answers.State consistency, plausibility, necessity, and transition payloads have family-specific filtering, sorting, deduplication, and rejection rules; recovery statuses and trace counts also require exact widths or integer types. Rate limits retry automatically, while other backend or transport failures are persisted for possible manual targeting with the unchanged prompt.Source: Answer extraction and normalization… · p. 16

A.8 Per-Task Family Complexity Curves

  • B1–B4 complexity bins are formed by ordering retained items within each task family by complexity and mechanism/task identifiers, then cutting at tie-block boundaries nearest equal-count targets.Equal-complexity items stay together, with exact distance ties assigned to the earlier boundary, producing near-equal quartiles.Source: §A.8 · p. 16

A.9 OpenModelica–Qualitative Consistency Check

  • The OpenModelica check freezes numerical observations before qualitative comparison and independently tests exact MDD state relations and directed transition paths.The comparison is task- and gold-independent but not a clean-room implementation; the bounded suite uses fixed Modelica instances and does not map isolated numerical samples to benchmark episode nodes.Source: §A.9 · p. 17
  • 341 distinct observed scenarios across 72 experiments yielded 339 direct agreements, two boundary-compatible states, and zero contradictions or inconclusive cases.The two boundary cases involved small positive velocities mapped to REST by the 10^-9 landmark tolerance, while exact MDD checks accepted MOVING_POSITIVE; the raw equations remained satisfied. Validation covers only behavior observed in the finite runs.Source: Table 8 · p. 17, §A.9 · p. 17

A.10 Model Inventory

  • The model inventory distinguishes documented agentic-capable, tool-capable, general, and reasoning-modifier classes, while the evaluated configurations use no agentic scaffold.Agentic-capable classification requires first-party evidence of closed-loop, multi-step tool use; reasoning, long context, or isolated function calling alone is insufficient.Source: §A.10 · p. 17, Table 9 · p. 18, Table 10 · p. 18
Loading 2609.34636v1…