AI scientists produce results without reasoning scientifically

Martiño Ríos-García; Nawaf Alampara; Chandan Gupta; Indrajeet Mandal; Sajid Mannan; Ali Asghar Aghajani; N. M. Anoop Krishnan; Kevin Maik Jablonka

Source-linked AI summary

AI scientists produce results without reasoning scientifically

Martiño Ríos-García, Nawaf Alampara, Chandan Gupta, Indrajeet Mandal, Sajid Mannan, Ali Asghar Aghajani, N. M. Anoop Krishnan, Kevin Maik Jablonka

arXiv:2604.18805v1Apr 20, 2026cs.AIcond-mat.mtrl-scics.LG

TL;DR

Whether LLM-based scientific agents reason according to self-correcting epistemic norms remains poorly understood, despite task-based evaluations. Across eight domains and more than 25,000 runs, this study finds that agents execute workflows but routinely fail to reason scientifically, with behavior driven mainly by the base model rather than the scaffold.

Problem
Whether LLM-based agents follow the epistemic norms that make scientific inquiry self-correcting remains poorly understood, limiting assessment of whether their conclusions are justified beyond task success.
Method
The study combines controlled performance evaluation with behavioral analysis of reasoning across three models, two scaffolds, and eight scientific domains.
Results
Across domains and scaffold conditions, evidence is ignored in 68% of traces, while agents show the same undisciplined reasoning in workflows and hypothesis-driven inquiry.
Takeaways & Limitations
Evaluating scientific agents requires direct assessment of reasoning processes because answer-based evaluation leaves their epistemic failures invisible.
Takeaways & Limitations
Attributing observed differences to the base model or scaffold assumes all configurations face identical tool interfaces and environment dynamics.

Abstract

from arXiv · show

Large language model (LLM)-based systems are increasingly deployed to conduct scientific research autonomously, yet whether their reasoning adheres to the epistemic norms that make scientific inquiry self-correcting is poorly understood. Here, we evaluate LLM-based scientific agents across eight domains, spanning workflow execution to hypothesis-driven inquiry, through more than 25,000 agent runs and two complementary lenses: (i) a systematic performance analysis that decomposes the contributions of the base model and the agent scaffold, and (ii) a behavioral analysis of the epistemological structure of agent reasoning. We observe that the base model is the primary determinant of both performance and behavior, accounting for 41.4% of explained variance versus 1.5% for the scaffold. Across all configurations, evidence is ignored in 68% of traces, refutation-driven belief revision occurs in 26%, and convergent multi-test evidence is rare. The same reasoning pattern appears whether the agent executes a computational workflow or conducts hypothesis-driven inquiry. They persist even when agents receive near-complete successful reasoning trajectories as context, and the resulting unreliability compounds across repeated trials in epistemically demanding domains. Thus, current LLM-based agents execute scientific workflows but do not exhibit the epistemic patterns that characterize scientific reasoning. Outcome-based evaluation cannot detect these failures, and scaffold engineering alone cannot repair them. Until reasoning itself becomes a training target, the scientific knowledge produced by such agents cannot be justified by the process that generated it.

1 Introduction

LLM-based scientific agents can complete scientific tasks, but their opaque reasoning does not reliably follow the evidence-based, self-correcting norms of scientific inquiry. Across evaluations of multiple models, scaffolds, and domains, the base model largely determines behavior while scaffold engineering contributes little.

Historical contrast: DENDRAL made every inference inspectable by generating candidates, testing spectral matches, and eliminating structures that failed.Its explicit rules provided epistemic machinery for scrutinizing each reasoning step.
Historical contrast: LLM-based agents derive conclusions from statistical regularities, leaving the epistemic process by which they reason largely inaccessible to scrutiny.Unlike DENDRAL, these systems do not show their reasoning at every step.
Evaluation gap: Task-completion benchmarks reveal whether scientific agents succeeded, but not how or why they succeeded or how they will behave on new problems.Existing evaluations include question answering, workflow execution, result reimplementation, and manuscript production judged by peer review.
Study design: The study combines systematic performance evaluation with behavioral analysis of the epistemological structure of agent reasoning.It uses controlled environments, standardized tools, graded scopes, and fine-grained diagnostics to separate base-model and scaffold contributions.
Main findings: Across 3 frontier models, 2 scaffold architectures, and 8 scientific domains, the base model primarily determines performance and behavior while scaffolds contribute little.Agents routinely ignore gathered evidence, commit to untested hypotheses, and fail to revise beliefs after contradictory data, whether executing workflows or conducting hypothesis-driven inquiry.

2 Results

Across eight scientific domains and more than 25,000 agent runs, performance depended primarily on the base model, while agents’ epistemic reasoning remained structurally unreliable. Agents executed workflows but frequently ignored evidence, failed to test or revise hypotheses, and became unreliable across repeated trials.

Benchmark design: 25,000+ agent runs across eight scientific domains benchmarked workflow execution and hypothesis-driven inquiry at varying scopes.The environments spanned procedurally complex workflows and tasks requiring hypothesis formation, discriminating experiments, and belief revision.
Performance determinants: Below 60% was the strongest configuration’s performance in broad-scope Inorganic Qualitative Analysis and Spectroscopic Structure Elucidation.Workflow tasks were reliable at guided scopes, but performance declined with broader scope and declined more steeply as epistemic demand increased.
Performance determinants: 41.4% of explained variance in task success came from reasoning ability, while scaffold effects accounted for only 1.5%.The latent factor model compared environment, scope, scaffold, verbosity, and IRT-derived abilities across eight candidate specifications, selecting M7 by PSIS-LOO cross-validation.
Epistemic reasoning: 53% of traces contained untested claims, while evidence non-uptake reached 82% in workflow domains, 66% in strategic reasoning, and 60% in hypothesis-driven domains.The same reasoning topology appeared across tasks despite different demands, including hypothesis-test cycles, combinatorial search, and simulate-validate-refine loops.
Epistemic reasoning: Higher-performing models retrieved more relevant knowledge and executed procedures more accurately, but their epistemic reasoning structure did not change correspondingly.Models differed in operation frequencies, with stronger models producing more hypothesis and evidence nodes, while reasoning-graph topology remained similar.
Reliability: Pass∧k dropped below 0.05 by k = 4–6 for Spectroscopic Structure Elucidation and Inorganic Qualitative Analysis under baseline conditions.The decay remained similar after early-step success-trace interventions, and ReAct and tool-calling agents showed similar dependence on late-stage successful steps.

3 Discussion

Current LLM-based agents approach ceiling performance in workflow-construction domains but fail to adapt their reasoning in hypothesis-driven inquiry, where evidence is frequently ignored. Their epistemic shortcomings persist across scaffold conditions and remain invisible to answer-only evaluation, making explicit measurement necessary.

3 Discussion: In workflow-construction domains, agents approach ceiling performance, but in hypothesis-driven domains they fail to adapt reasoning to problem demands.Evidence is ignored in 68% of traces.
3 Discussion: The documented reasoning patterns persist across all tested scaffold conditions, despite engineering efforts focused on prompting, orchestration, and tool interfaces.The passage links this persistence to uncertainty about the reliability of answers reached while contradictory evidence is ignored and hypotheses remain untested.
3 Discussion: Explicit measurement is required to characterize agents’ epistemic behavior because, unlike human scientists, they lack institutional constraints such as peer review, replication, and reputational consequences.The evaluation framework measures epistemic behavior across tasks and configurations.
3 Discussion: Answer-only evaluation leaves agents’ non-scientific epistemic behavior invisible, allowing it to shape the knowledge they help produce.The analysis concludes that these agents do not exhibit the patterns that characterize scientific reasoning.

4 Methods … A.4 Item response theory for model evaluation

The paper formalizes scientific agents, evaluates them across diverse environments, and combines outcome scoring with trace annotation and latent-capability modeling. It situates Corral against outcome-focused benchmarks and AI-scientist systems while using IRT to separate model ability from item difficulty.

4.1 Formalism: Agents implement a composite policy combining base-model token choices with scaffold-controlled prompting, tool routing, memory, and orchestration within a text-based decision process.The state contains the task prompt and conversation history, while tools belong to the environment rather than the agent.
4.2 Task and tool formalism: Corral represents benchmark items as individual tasks or dependency graphs, constructs controlled prompts and tool descriptions, and varies tool verbosity independently for ablation.Task groups inject predecessor outputs sequentially; verbosity levels range from brief to comprehensive.
4.3 Agent implementations: Experiments vary only model backend and scaffold across eight domain environments whose tools, observations, task descriptions, and scoring functions support workflow execution and scientific inquiry.The environments include spectroscopy, qualitative analysis, circuit inference, retrosynthesis, AFM, molecular simulation, adsorption-surface construction, and ML-based property prediction.
4.5 Manual trace annotation: Manual trace annotation and diagnostic question design provide process-level and capability-level measurements, with expert-reviewed knowledge and reasoning items and 773 selected traces annotated by domain experts.Epistemological graphs encode hypotheses, tests, evidence, judgments, updates, commitments, and their relations; productive patterns and anti-patterns are defined structurally.
4.7 IRT and phenomenological model: The two-stage model first estimates standardized knowledge and reasoning abilities with separate 2PL IRT models, then predicts balanced benchmark success using Bayesian Binomial models with scaffold, verbosity, and scope effects.Candidate models M1–M8 are compared with PSIS-LOO, and M7 is selected as best-fitting because its knowledge and reasoning slopes vary by environment × scope.
4.9 Trace intervention experiment: Trace interventions test whether performance reflects latent capability or accumulated conversational context, while token-level log-probabilities quantify confidence across environments.Intervention runs use temperature 0.7, unlike the deterministic main evaluation at temperature 0.0; more negative mean log-probabilities indicate greater uncertainty.
A.1 Benchmarks for scientific agents: Corral extends outcome-based scientific-agent benchmarks by evaluating how results are produced, addressing systems that conduct end-to-end research without a shared instrument for reasoning quality.Existing evaluations treat lookup-based and hypothesis-tested correct answers equivalently, while Corral supplies reusable environments, tools, scoring functions, and a runner.
A.3 Scaffolds and the base-model/scaffold attribution: The paper examines attribution between base-model competence and scaffold orchestration and applies IRT to estimate latent capability per model–environment pair separately from item difficulty.This distinction is presented as an alternative to relying on aggregate Pass@k alone.

A.5 Epistemic evaluation of AI in science … C.1 Prompt construction

The paper frames epistemic evaluation as necessary for assessing whether AI-generated scientific claims are justified, while documenting the rapid growth of AI-scientist research and specifying Corral’s evaluation framework, terminology, and prompt construction. Corral separates agents from environments and holds most prompt components constant so configurations can be evaluated systematically.

A.5 Epistemic evaluation of AI in science: Process-level evaluation addresses whether AI-generated scientific claims are epistemically justified, rather than treating predictive accuracy as sufficient.The behavioral analysis represents reasoning traces as graphs and detects structures associated with productive inquiry, including Popperian falsification.
A.6 Rise of AI Scientists: AI-scientist publications have risen sharply within chemistry and materials AI literature, reaching the upper 10^2 range in the most recent partial year.The broader AI-for-chemistry literature is described as reaching roughly 10^4 papers per year, while the strict AI-scientist subset grew from about 10^1 papers in 2018.
A.6 Rise of AI Scientists: The AI-scientist subset rose from about 1% of AI-for-chemistry output through 2024 to roughly 2% in 2025 and nearly 5% in the current partial year.Despite this growth, it represented approximately 0.01% of all chemistry and materials science publishing through 2024 and around 0.06% in 2025.
B Glossary: The benchmark glossary defines domains, scopes, environments, tasks, subtasks, configurations, trials, and traces as linked units of agent evaluation.An environment is a concrete domain instance exposing tools, a task description, and a scoring function; a trace records the conversation history from a trial.
B Glossary: The glossary distinguishes Pass@k, requiring at least one successful trial, from Passk, requiring every independent trial to succeed.It also defines epistemic operations and productive or breakdown motifs as graph-based annotations of reasoning traces.
C Corral: Corral separates environments from agents through a standard HTTP interface, while its runner iterates trials, checkpoints state, and aggregates results.Agents access task guidance, tools, and model responses through the router, and environments expose task descriptions, tools, and scoring through RESTful endpoints.
: Framework: Corral’s framework treats tool docstrings as part of the environment-agent contract and filters their tagged sections by documentation verbosity.The two evaluated scaffolds are ReAct and structured tool calling.
C.1 Prompt construction: The base-model message sequence combines a static system prompt, environment-specific task prompt, scaffold template, and tool descriptions, with only the task prompt varying across environments.ReAct injects textual tool guidance inline, whereas structured tool calling passes schemas through the native function-calling API.

C.2 Example of different verbosity · D Domains and environments

The paper illustrates verbosity control by progressively retaining tagged sections in tool documentation, then describes Corral as a framework spanning eight scientific domains and multiple task-specific environments. These environments expose tools, task descriptions, scoring functions, and domain-specific evaluation infrastructure.

C.2 Example of different verbosity: At each verbosity level, tool documentation retains a cumulative subset of tagged sections.The tagged sections define the content authored for each tool’s documentation.
C.2 Example of different verbosity: The least detailed tool description specifies searching the NMRShift database for entries matching or chemically similar to a SMILES string.It also identifies the smiles and top_k arguments and returns relevant database entries.
C.2 Example of different verbosity: More verbose documentation adds usage guidance, vector-search mechanics, chemical embeddings, workflow integration, and limitations.The described search uses the nmrshiftdb2 collection and MoLFormer-XL-both-10pct embeddings, with results limited by top_k.
C.2 Example of different verbosity: The expanded example demonstrates a search_by_smiles("CCO") call and explains that results are dictionaries sorted by similarity score.The function requires valid SMILES input and defaults top_k to 10.
D Domains and environments: Corral environments provide task descriptions, tools, and scoring functions through a server interface that agents access with HTTP requests.The runner handles checkpointing, aggregation, and metric computation.
D Domains and environments: Each of the eight domains contains multiple task scopes, with each scope implemented as a separate environment.Extended Data Figure 1 summarizes the agent input and produced artifact for each domain.
D Domains and environments: Table D.3 characterizes each domain by difficulty levels, tasks per level, available domain-specific tools, typical trace length, and backing engine, instrument, dataset, or library.The table defines trace length as the mean number of agent-generated messages, excluding system and initial user messages.

D.1 Inorganic Qualitative Analysis Environment Example · E Capability assessment items

The inorganic qualitative analysis environment asks agents to identify inorganic cations through tool-based experiments. Capability profiles are estimated from environment-specific question-answer items divided into knowledge and multi-step reasoning sets.

D.1 Inorganic Qualitative Analysis Environment Example: 15 mL of an unknown sample solution contains 3 different inorganic cations that agents must analyze.The task requires performing tests and experiments with provided tools and reagents.
D.1 Inorganic Qualitative Analysis Environment Example: Agents perform tests and experiments on the sample using the environment’s available tools and reagents.The workflow overview identifies tool-based interaction, including pH measurements and flame tests.
D.1 Inorganic Qualitative Analysis Environment Example: The workflow has three phases: initialization, iterative interaction, and a final response stage.Initialization constructs the prompt; iterative interaction enables tool calls and external information access.
E Capability assessment items: Each environment’s items are partitioned into knowledge and reasoning sets.The knowledge set probes domain-specific facts, while the reasoning set requires multi-step inference over domain concepts.
E Capability assessment items: Tables E.4 and E.5 provide representative knowledge and reasoning items for each environment.The examples include one representative item of each type per environment.

E.1 Item generation protocol … G.5 Binomial modeling

The paper evaluates scientific agents with expert-reviewed knowledge and reasoning benchmarks, annotated execution traces, aggregate and skill-level performance analyses, behavioral markers, IRT capability estimates, and hierarchical binomial variance modeling. Across these analyses, model capability dominates scaffold effects, while performance declines as epistemic demand increases and differs across domains and reasoning capabilities.

E.1 Item generation protocol: Items are expert-authored, divided into knowledge and reasoning categories, and reviewed by two independent experts for correctness, clarity, originality, scope, difficulty, diversity, and metadata.Knowledge items test factual or procedural understanding; reasoning items require multi-step inference, scenario analysis, or quantitative problem-solving.
F Marker manual annotation: 773 traces were annotated end-to-end by domain experts using a navigable trace tool and a controlled taxonomy of behavioral markers, with optional step-level and trace-level comments.Each environment had one assigned expert reviewing accepted traces across models and ReAct or tool-calling configurations.
G Performance analysis: The base model is the primary determinant of agent performance, while scaffold, tool-documentation verbosity, and item-level factors explain smaller shares of variance.Aggregate scores, per-skill performance, hierarchical Bayesian variance decomposition, and task-level calibration are used in the performance analysis.
G.1 Aggregate scores: Claude Sonnet 4.5 leads in most environments, GPT-4o follows, and ReAct and Tool-Calling agents perform comparably on average.Model gaps narrow for lower-complexity S1 scopes and tightly planned environments, while scaffold differences become more pronounced at higher scopes and in workflow-construction environments.
G.2 Subtask-category gradient: Reasoning and validation scores are generally lower and more variable than retrieval and execution scores across model–agent combinations.These categories require logical inference, consistency checking, and hypothesis testing, whereas retrieval and execution test more routine information processing.
G.3 Manual marker annotation: Claude Sonnet 4.5 produces substantially more explicit reasoning steps than GPT-4o across both scaffolds, including 392 vs. 69 under ReAct.ReAct also elicits more planning_statement annotations for Claude Sonnet 4.5 than tool calling, with 240 vs. 34 annotations.
G.4 Latent factor modelling: Latent knowledge and reasoning capabilities are estimated separately for each model–environment pair using two-parameter logistic item response theory models.The QA items target domain-specific factual knowledge and multi-step reasoning, with posterior means and standard deviations estimated via MCMC.
G.5 Binomial modeling: Hierarchical binomial modeling decomposes agent-success variance between model capability and design choices under explicit interaction and grouping assumptions.Models are selected with PSIS-LOO, balancing predictive accuracy and complexity; the main-text variance decomposition and posterior predictive checks use the top-ranked model.

G.5.1 Model specifications · G.6 Intervention experiments

The model-specification analysis identifies M7 as the selected best-fitting model and finds that model identity dominates explained variance while scaffold and tool-description effects are comparatively small. Intervention experiments replay partial successful or failed traces under controlled conditions to test whether prior reasoning steers subsequent agent performance.

G.5.1 Model specifications: M7 is the selected latent factor model, with common sum-to-zero effects and a full weakly informative prior specification.The common notation is summarized in Table G.11, while Table G.12 gives M7’s priors, soft-centering constraints, and non-centered hierarchical parameterization.
G.5.1 Model specifications: R^2 > 0.95 confirms that M7’s hierarchical predictions generalize beyond in-sample fit and recover per-task difficulty after accounting for abilities, scaffolds, and tool-description verbosity.Predictive fidelity is assessed with PSIS approximating leave-one-out predictions, whose probabilities tightly cluster around the identity line.
G.5.1 Model specifications: 45.2% [41.4, 48.8] of explained variance is attributed to model identity in M7, exceeding the next-largest environment × scope component at 32.5%.The model-identity component is driven by IRT-derived knowledge and reasoning abilities, and its credible interval does not overlap the next-largest contributor.
G.5.1 Model specifications: Less than 2% of variance is jointly explained by scaffold choice and tool-description verbosity in M7, while scaffold, verbosity, and task category remain below 4% combined across specifications.Across M1–M8, model identity contributes 31% in M1 to 46% in M8, consistently ranking first or second.
G.5.1 Model specifications: Higher reasoning ability predicts better task performance across all 16 environment × scope cells, with total reasoning slopes ranging from 0.48–5.76.The reported slopes are positive across every environment × scope cell.
G.6 Intervention experiments: The intervention protocol samples successful or failed traces from mixed-outcome tasks, injects selected assistant turns at k ∈{1, 2, −2, −1}, and re-executes tool calls in the live environment.The agent resumes autoregressively from the seeded history and is unaware that the initial turns were replayed.
G.6 Intervention experiments: 15 trials per condition compare baseline, success intervention, and failed intervention using Pass@k and Pass∧k for k = 1, . . . , 15.Failed-trace injections serve as a negative control for whether successful reasoning genuinely aids recovery, with a different trace sampled uniformly for each trial.
G.6 Intervention experiments: Hypothesis-driven inquiry environments show minimal benefit from early successful steps, often at or below baseline, before a sharp jump later in the trajectory.Figure G.12 reports distinct recovery profiles for successful and failed trace interventions across environments.

G.7 Token-level log-probability analysis

Mean token-level log-probability is used as an environment-level proxy for model uncertainty, with workflow-construction domains showing higher confidence than hypothesis-driven and strategic-reasoning domains.

Method: Mean token-level log-probability serves as an environment-level proxy for model uncertainty.The analysis uses mean token-level log-probabilities as defined in Section 4.10.
Method: Top-1 token log-probabilities are pooled across assistant messages and trials, excluding special control tokens, to compute each environment’s mean ¯ℓe.The resulting per-environment means are reported in Figure G.14.
Results: Hypothesis-driven environments yield consistently lower mean log-probability than workflow-based environments.This ordering mirrors the reported gradient in epistemic demand.
Results: Workflow-construction domains have the least negative means (¯ℓe ≥−0.22), indicating high token-level confidence consistent with well-defined solution paths.The domains are ML-based Property Prediction, Adsorption Surface Construction, and Molecular Simulation.
Results: Hypothesis-driven and strategic-reasoning domains share the most negative values (¯ℓe = −0.27).These domains are Inorganic Qualitative Analysis, Spectroscopic Structure Elucidation, and Retrosynthetic Planning.

G.8 Verbosity detailed results

Tool-documentation verbosity has a negligible effect on agent scores when model, scaffold, environment, and scope are held fixed. Scores remain close across verbosity levels, with differences within 0.05 points in every environment and verbosity explaining <0.1% of variance.

Scores were compared across three tool-documentation verbosity levels while holding model, scaffold, environment, and scope fixed.Scores were aggregated per environment, model, and scaffold.
<0.1% of variance was attributed to verbosity by the latent factor model.This is consistent with verbosity having a negligible effect on scores.
Within 0.05 points, workflow and comprehensive scores differed from brief verbosity in every environment.Model-level scores were also close, and the two scaffolds tracked each other across verbosity levels.

H Reasoning process analysis … H.3 Pattern taxonomies

The paper analyzes agent reasoning as epistemic graphs, using a two-stage annotation pipeline to identify productive motifs and reasoning breakdowns. Across models and environments, breakdowns consistently outweigh disciplined inquiry patterns, while prompt optimization improves annotation agreement.

H Reasoning process analysis: Each reasoning trace is represented as an epistemic-operation graph, allowing structural templates to detect productive patterns and reasoning breakdowns independently of answer correctness.The analysis focuses on how agents reach answers rather than whether those answers are correct.
H Reasoning process analysis: 88% of traces show evidence non-uptake for both Claude Sonnet 4.5 and GPT-4o, while untested claims occur in 66% and 62%, respectively.Convergent multi-test evidence and evidence-guided test redesign each occur in at most 1% of traces; refutation-driven belief revision occurs in 4% and 1%.
H Reasoning process analysis: Reasoning breakdowns exceed productive motifs at every environment level, indicating that undisciplined reasoning appears across task types rather than only at the highest difficulty.This prevalence is shown in Figure H.17.
H.1 LLM annotation pipeline: The epistemological graph prevalences are generated by a two-stage annotation pipeline applied to each completed agent run’s serialized message history.The pipeline operates on the serialized message history of each completed run.
H.1 LLM annotation pipeline: Stage 1 labels message-level epistemic operations, and Stage 2 constructs directed, typed edges between related nodes using fixed vocabularies and supporting quotes.Both stages use overlapping 20-message windows with a stride of 15 and Claude Sonnet 4.5 at temperature 0.7.
H.2 Annotations prompts: Prompt optimization raised human-human PABAK agreement from 85.8% to 92.6% and human-LLM percent agreement from 89.3% to 95.7%.The improvements were similar for node and edge annotations.
H.3 Pattern taxonomies: Pattern taxonomies define productive motifs and reasoning breakdowns as explicit structural templates over annotated graphs, organized around hypothesis, evidence, and inquiry control.Templates specify required node types, edge types, and configurations.
H.3 Pattern taxonomies: Productive motifs include refutation-driven belief revision, convergent multi-test evidence, evidence-guided test redesign, and hypothesis reranking, while breakdowns include untested claims, evidence non-uptake, unsupported judgments, and stalled revision.The taxonomy also includes fixed belief traces, disconnected evidence, premature commitment, and uninformative tests.

H.4 Detailed annotation results

Across models and environments, reasoning breakdowns are more prevalent than productive epistemic motifs. This imbalance persists across environment types and task scopes, with agents showing no systematic adaptation as complexity increases.

Per-model results: Breakdowns consistently exceed productive motifs across all models, confirming the aggregate prevalence trend.Table H.16 reports model-level prevalence averaged across environments and separates hypothesis handling, evidence handling, and inquiry control.
Environment-level prevalence: Reasoning breakdowns outnumber productive motifs across every environment level and type.Hypothesis-driven environments have the highest overall annotation density, consistent with their greater demands for iterative belief revision.
Scope invariance across subtask levels: Despite a fourfold increase in task complexity from S1 to S4, epistemic operation prevalence and ordering do not shift systematically.The analysis averages across multi-level environments and covers hypothesis handling, evidence handling, and inquiry control.
Scope invariance across subtask levels: Agents do not systematically adapt their epistemic behavior to broader or more complex problem settings.This conclusion follows from the observed invariance in both absolute prevalence and relative ordering across increasing problem scopes.

H.5 Annotation agreement analysis

Validation by three domain experts found substantial human–human agreement and 95.7% human–LLM agreement, exceeding the 92.6% human–human baseline. Agreement was higher for node-level than edge-level annotations, while low Cohen’s κ reflected class imbalance.

Validation design: Three domain experts independently reviewed 25 representative traces from 626 total traces across annotation rounds before and after prompt optimization.Agreement was evaluated at both human–human and human–LLM levels.
Human–human agreement: 92.6% overall human–human agreement corresponded to a mean PABAK of 0.853, indicating substantial agreement.Node-level PABAK reached 0.976, whereas edge-level PABAK ranged from 0.737 to 0.913.
Human–human agreement: 0.067 overall Cohen’s κ was low because the label distribution was heavily skewed toward the correct class.PABAK was additionally reported to adjust for prevalence and bias.
Human–LLM agreement: 95.7% overall human–LLM agreement exceeded the 92.6% human–human baseline on optimized annotations.Node agreement ranged from 92.8% to 99.3%, while edge agreement ranged from 87.1% to 99.8%.

H.6 Other agent behavior analysis · H.7 Illustrative traces

Behavioral analyses show that model choice shapes tool-use and output intensity more strongly than scaffold choice, while illustrative traces reveal recurring failures to use evidence, test hypotheses, revise beliefs, and repair contradictions. These failures appear in both scientific workflows and hypothesis-driven inquiry, including cases where agents ignore correct candidate structures, assert untested interpretations, or preserve beliefs after repeated errors.

H.6 Other agent behavior analysis: ReAct and tool-calling agents show similar action profiles, whereas model differences more strongly influence which tools are invoked in specific environments.The analysis examines tool actions, output verbosity, and tool calls to distinguish scaffold effects from model-dependent interaction strategies.
H.6 Other agent behavior analysis: Models differ more in output length and tool-use intensity than in action mixtures across environments.Verbosity varies substantially by environment, and models differ in tokens produced and tools called; these differences are stronger than action-mixture differences.
H.6 Other agent behavior analysis: Despite ReAct’s explicit Thought category, token distributions across agents are similar, while Resistor produces markedly more tokens per message than other environments.Environments with lower performance tend to produce more tokens, according to the detailed output-token analysis.
H.7 Illustrative traces: Representative traces encode hypotheses, tests, evidence, judgments, and commitments as nodes, marking missing edges that should occur in sound reasoning chains.The trace annotations also identify edges linking epistemic nodes and provide supporting annotator quotes for each breakdown category.
H.7 Illustrative traces: A correct structure among 20 returned candidates is ignored, as the agent guesses structures sequentially instead of using the isomer evidence to constrain hypotheses or judgments.The candidate-list node has zero outgoing informs edges, and the agent shifts to a tangential DBE calculation before guessing alternatives.
H.7 Illustrative traces: A detailed NMR interpretation receives no test, as the agent asserts coupled CH2 groups and an acetyl group before requesting mass spectrometry.The interpretation node has zero outgoing tests edges, so the proposed structural features are not directly evaluated.
H.7 Illustrative traces: 6 hypothesis nodes and zero updates_to edges characterize a trace in which repeated LAMMPS failures do not prompt revision of earlier beliefs.After mistaking /potentials/SW for a file and then attributing lost atoms to heating, both beliefs persist unmodified through the remaining 15 messages.
H.7 Illustrative traces: Despite a 3H-versus-6H NMR discrepancy, the agent commits to the isopropyl ester after dismissing the contradiction as a simulation artifact.The contradiction edge from the simulated evidence to the hypothesis has no accompanying updates_to edge, and the final commitment remains unresolved.

I Scope and limitations

The evaluation’s conclusions are bounded by independent-task episodes, minimal scaffolds, limited model coverage, and practical compute and measurement constraints. Reported results therefore describe systems under a common minimal scaffold rather than their best-case performance with model-specific hardening.

Scope of the evaluation: Each task is an independent episode, so agents cannot carry knowledge, strategies, or refined heuristics across tasks.The evaluation uses ReAct and native tool calling with a flat append-only history and simple context truncation.
Model coverage and compute: 3 models—Claude Sonnet 4.5, GPT-4o, and GPT-OSS (gpt-oss-120b)—were evaluated, while other prominent model families were omitted.The models were selected to span proprietary and open-weight systems at the current capability frontier.
Model coverage and compute: Approximately 3 billion tokens across 138 configuration-environment pairs at five trials each were consumed, with an estimated $7.5k API cost for the two proprietary models.The estimate excludes infrastructure costs for the self-hosted open-weight model.
Measurement caveats: 41% of GPT-OSS-120B traces were affected by malformed responses, averaging 1.26 errors per trace; GPT-4o averaged 0.28 errors per trial, affecting 6.2% of trials.Log-probability analyses are restricted to GPT-OSS because proprietary providers do not expose the required per-token log-probabilities.
Scope and limitations: The reported numbers reflect each system under the same minimal scaffold, not best-case performance with model-specific hardening.The study notes that deliberate design choices and practical constraints bound comparisons with future work using richer scaffolds, broader model coverage, or larger compute budgets.