AI scientists produce results without reasoning scientifically

• Unlike DENDRAL’s inspectable, evidence-matching inferences, LLM agents’ statistically learned reasoning is largely inaccessible to scrutiny.DENDRAL generated candidate molecular structures from mass spectra and eliminated those that failed to match the evidence.Source: §1 · p. 2
• Because LLM-based agents direct scientific inquiry, how they reason matters for whether they produce understanding beyond correct predictions.The introduction notes that machine-learning adoption has narrowed the hypotheses scientists pursue, while LLM agents direct inquiry itself.Source: §1 · p. 2
• Task-completion benchmarks show whether an agent succeeds, but not how or why, and do not predict its behavior on new problems.The underlying epistemic process bears on whether results are justified and whether agents can be trusted beyond tested conditions.Source: §1 · p. 2
• The study combines controlled performance evaluation that separates base-model and scaffold contributions with behavioral analysis of agents’ epistemological reasoning.It evaluates three frontier models with two scaffold architectures across eight scientific domains using graded task scopes and fine-grained subtask diagnostics.Source: §1 · p. 2
• The base model primarily determines performance and behavior, while agents ignore gathered evidence, commit to hypotheses without testing, and fail to revise beliefs after contradiction.The same undisciplined reasoning appears in computational workflows and hypothesis-driven inquiry, and the authors argue that evaluation must assess reasoning directly.Source: §1 · p. 3

2 Results

• The benchmark spans eight domains from procedural workflow execution to hypothesis-driven inquiry, with task scope graded within domains.Tasks include workflow construction, strategic retrosynthesis, and inquiry requiring hypotheses, discriminating experiments, and belief revision; environments also isolate operations such as retrieval, execution, reasoning, and validation.Source: §2 · pp. 3, 5

The base model determines performance

• Reasoning ability accounted for 41.4% of explained task-success variance, compared with 1.5% for scaffold choice.Across environments, performance declined as scope broadened and epistemic demand increased; even the strongest configuration scored below 60% in the broadest Inorganic Qualitative Analysis and Spectroscopic Structure Elucidation tasks.Source: The base model determines performance · pp. 5, 7, Figure 3 · p. 7

Epistemological structure of agent reasoning

• Evidence was not incorporated in 68% of traces, beliefs were never updated in 71%, and only 26% showed refutation-driven revision.Untested claims appeared in 53% of traces overall and 63% in hypothesis-driven domains; convergent multi-test evidence occurred in just 7%, despite evidence-led hypothesis generation in 65%.Source: Epistemological structure of agent… · p. 8
• Agents used similar reasoning structures across domains and models, with evidence non-uptake at 82% in workflow, 66% in strategic, and 60% in hypothesis-driven domains.Convergent multi-test evidence remained at or below 10% in every domain group, and increasing task scope did not shift epistemic operations. Stronger models produced more hypothesis and evidence steps, but their reasoning structure did not change correspondingly.Source: Epistemological structure of agent… · pp. 8, 10

Trace interventions and reliability

• Near-complete successful traces were needed to improve performance in hypothesis-driven and strategic tasks, while early successful steps could help workflow tasks.Under baseline conditions, the probability that all repeated trials succeeded fell below 0.05 by k = 4–6 in Spectroscopic Structure Elucidation and Inorganic Qualitative Analysis; early-step success-trace interventions showed similar decay. Both scaffold architectures showed similar patterns.Source: Trace interventions and reliability · p. 10, Figure 5 · p. 11

3 Discussion

• Workflow agents approach ceiling performance, but in hypothesis-driven tasks evidence is ignored in 68% of traces, refutation-driven revision occurs in 26%, and convergent multi-test evidence is rare.These patterns persist across scaffold architectures, domains, and near-complete successful-trajectory interventions, and are properties of the base model.Source: §3 · p. 12
• Because the process determines whether knowledge is justified, scaffold-only fixes are unlikely; addressing these patterns will likely require training base models using epistemic criteria.The framework provides reproducible tasks, tools, and trajectory scoring that can define training signals for reasoning processes.Source: §3 · p. 12
• The evaluation framework explicitly measures epistemic behavior across tasks and configurations, and supports community contributions of new environments, models, scaffolds, and metrics.Its methods and findings extend to any setting where reasoning processes determine the value of results.Source: §3 · p. 12
• Unlike human scientists, who operate under peer review, replication, and reputational consequences, LLM-based agents face no such constraints; AI assistance may also erode independent problem-solving [34].The authors argue that characterizing agent epistemic behavior therefore requires explicit measurement.Source: §3 · p. 12
• When agents are evaluated only by their answers, their divergence from scientific reasoning remains invisible and shapes the knowledge they help produce.The analysis finds that their epistemic behavior does not exhibit patterns that characterize scientific reasoning.Source: §3 · p. 12

4.1 Formalism; 4.2 Task and tool formalism

• The LDP framework treats each agent as a composite of model and scaffold policies acting over text-based task prompts and conversation histories.Tools belong to the environment, whose standardized interfaces and dynamics are shared across agent configurations; task groups are sequenced by dependencies, and tool-description verbosity is varied independently of task prompts.Source: §4.1 · p. 13, §4.2 · pp. 13–14

4.3 Agent implementations; 4.4 Domains and environments

• Corral experiments varied three model backends and two scaffolds, with domain environments supplying tools, task descriptions, and scoring functions.Main-text runs used temperature 0.0 and brief tool descriptions; environments included spectroscopic analysis, inorganic identification, circuit inference, and retrosynthetic planning.Source: §4.3 · p. 14, §4.4 · pp. 14–15

4.5 Manual trace annotation; Appendix

• Manual trace annotation labeled behavior-relevant nodes with positive, failure, or neutral markers, and expert annotators produced final records for 773 traces.The appendix lists detailed annotation results, agreement analysis, other behavior analyses, and illustrative traces.Source: §4.5 · pp. 16–17, Appendix · p. 31

4.6 Diagnostic question-answer pairs

• Diagnostic question banks separate domain knowledge from required reasoning operations, with expert-reviewed items evaluated for all three models at temperature 0.0.Knowledge banks ranged from about twenty to over one hundred questions per environment; reasoning banks contained twenty to twenty-five questions each.Source: §4.6 · p. 17

4.7 IRT and phenomenological model

• A two-stage model estimates latent knowledge and reasoning abilities with separate 2PL IRT models, then predicts benchmark success using Bayesian binomial models; M7 was selected by PSIS-LOO.The abilities are standardized before entering Stage 2, where model, environment, scaffold, scope, verbosity, and task-related factors are modeled; M7 allows knowledge and reasoning slopes to vary by environment × scope.Source: §4.7 · pp. 17–18

4.8 Epistemological graphs

• Epistemological graphs encode hypotheses, tests, evidence, judgments, updates, and commitments, with expert review finding 95.7% human–LLM agreement.Automated checks verified quote and edge validity; structural templates defined productive patterns and anti-patterns, including ignored evidence and unresolved contradictions.Source: §4.8 · pp. 18–19

4.9 Trace intervention experiment; 4.10 Token-level log-probability analysis

• Trace interventions replayed successful or failed prior trajectories at early or near-complete stages, while token confidence was estimated from pooled assistant-message log-probabilities.Intervention trials used temperature 0.7 and differed from deterministic main evaluation runs; more negative mean log-probabilities indicate greater uncertainty.Source: §4.9 · p. 19, §4.10 · pp. 19–20

A.1 Benchmarks for scientific agents; A.2 AI scientists; A.3 Scaffolds and the base-model/scaffold attribution; A.4 Item response theory for model evaluation

• Corral addresses outcome-only benchmarks by adding reasoning diagnostics, supports reusable agent-evaluation and training environments, and applies IRT to separate ability from item difficulty.The related work also frames orthogonal model-and-scaffold variation as a way to attribute their contributions to agent capability.Source: §A.1 · p. 31, §A.2 · p. 32, §A.3 · p. 32

A.5 Epistemic evaluation of AI in science

• Corral’s behavioral analysis annotates reasoning traces as operation graphs to detect productive inquiry patterns and failures such as ignored evidence or unresolved contradictions.This evaluates how agents reach answers, rather than only whether they reach the right answer.Source: §A.5 · p. 33

A.6 Rise of AI Scientists

• The broader AI-for-chemistry literature reaches roughly 104 papers per year, while the AI-scientist subset grew from about 101 papers in 2018 to the upper 102 range in the latest partial year.The subset already exceeds earlier full-year totals, though it remains much smaller than AI-for-chemistry research overall.Source: §A.6 · p. 34, Figure A.1 · p. 34
• AI-scientist papers rose from about 1% of AI-for-chemistry output through 2024 to roughly 2% in 2025 and nearly 5% in the current partial year.They represented approximately 0.01% of all chemistry and materials science publishing through 2024 and around 0.06% in 2025.Source: §A.6 · p. 34, Figure A.1 · p. 34

: Framework

• Corral separates environments from agents through an HTTP interface, while its runner iterates trials, checkpoints intermediate state, and aggregates configuration scores and diagnostics.Agents access task guides, execute tools, and route messages to a configured base model through the framework’s router.Source: : Framework · pp. 36, 38

C.1 Prompt construction

• Each environment’s task prompt follows a shared template covering the task, scientific goal, submission format, and available inputs, including predecessor artifacts for task groups.Some environments add a brief role preamble or workspace instructions when filesystem tools are available.Source: §C.1 · p. 39
• ReAct embeds a tool guide and requires XML-tagged thought–action–observation turns, whereas structured tool calling passes schemas through the native API and uses a “Final Answer:” prefix.Tool descriptions vary by verbosity and are injected inline for ReAct or through the function-calling API for tool calling.Source: §C.1 · p. 39
• Only task prompts vary across environments; system prompts, scaffold templates, and tool-injection mechanisms remain constant, and each message sequence combines four independently authored components.The system prompt is a static instruction establishing the agent’s role and context-management directive.Source: §C.1 · pp. 39–40

C.2 Example of different verbosity

• At each verbosity level, tool docstrings retain a cumulative subset of tagged sections, illustrated with search_by_smiles and listed in Table C.1.Source: §C.2 · p. 40, Table C.1 · p. 40
• The compact search_by_smiles description states that it searches NMRShift for matching or chemically similar compounds, accepts a SMILES string and optional top_k, and returns relevant entries.Source: §C.2 · p. 40
• Expanded documentation explains when to use the search, its vector-database and chemical-embedding approach, and how retrieved entries can support validation or comparison.The example also outlines a workflow from supplying a SMILES string to using retrieved entries; it mentions get_formula_from_smiles for formula validation.Source: §C.2 · p. 41
• The example notes that search errors can arise from embedding failures or a nonpositive top_k, while results are limited in number and embeddings may miss some chemical similarities.Source: §C.2 · pp. 42–43
• The fuller example specifies valid SMILES and positive top_k inputs, describes similarity-ranked results, and shows example structures, values, and a returned entry.Source: §C.2 · pp. 41, 43

D Domains and environments

• Across eight domains, each task scope is a separate environment; Table D.3 summarizes difficulty levels, tasks per level, available tools, typical trace length, and backing.The trace length is the mean number of agent-generated messages, averaged across models, agent architectures, and verbosity settings.Source: D Domains and environments · p. 47, Table D.3 · p. 47

D.1 Inorganic Qualitative Analysis Environment Example

• The inorganic-analysis workflow begins by prompting the agent with the task, available tools and arguments, and required response format.Source: Figure D.3 · p. 48
• During iterative interaction, the agent uses tools such as pH measurement and flame tests to perform experiments.Source: Figure D.3 · p. 48
• The agent can consult external information, including possible ions, reagent properties, color indicators, and current inventory.Source: Figure D.3 · p. 48
• At the end, the agent lists ions in the sample, and the environment returns a binary score based on comparison with ground truth.Source: Figure D.3 · p. 48

E Capability assessment items

• Latent capability profiles are estimated from environment-specific banks of curated question-answer items.Source: E Capability assessment items · p. 48
• Each environment’s items are partitioned into a knowledge set probing domain facts and a reasoning set requiring multi-step inference over domain concepts.Source: E Capability assessment items · p. 48
• Tables E.4 and E.5 show representative items of each type per environment, while Table E.6 reports per-model scores on the full bank.Source: E Capability assessment items · pp. 48–49

E.1 Item generation protocol

• Items were divided into factual-knowledge and multi-step reasoning questions, reviewed by two independent experts, and rejected or revised against a rubric.Claude Sonnet 4.5 won or tied on 13 of 16 environment–question-type pairs; reasoning questions were generally harder, and spectroscopic structure elucidation was the hardest domain for all three models.Source: §E.1 · pp. 49, 50, 54, Table E.6 · p. 54

F Marker manual annotation

• Experts annotated 773 execution traces using a controlled taxonomy, assigning behavioral markers step by step and adding optional comments.One expert covered each environment, reviewing accepted GPT-4o and Claude Sonnet 4.5 traces across ReAct and tool-calling configurations.Source: F Marker manual annotation · p. 55, Figure F.4 · p. 55

G Performance analysis

• The base model was the primary determinant of agent performance; scaffold, documentation verbosity, and item-level factors accounted for smaller shares of variance.The analysis also examined aggregate scores, per-skill performance, latent knowledge and reasoning, conversation-history interventions, and documentation-verbosity ablations.Source: G Performance analysis · p. 57

G.1 Aggregate scores

• Claude Sonnet 4.5 achieved the highest scores in most environments, winning or tying on 13 of 16 environment–question-type pairs; across agent types, it led in most environments, while ReAct and Tool-Calling agents performed comparably on average.Task-level cells average complete end-to-end trial episodes within each environment–scope bin; subtask scores average constituent steps and are generally higher because partial credit accumulates even when the overall task is unsolved.Source: §E.1 · p. 54, §G.1 · p. 57

G.2 Subtask-category gradient

• Across most model–agent combinations, retrieval and execution scores exceeded reasoning and validation scores, whose results also varied more across configurations.Both scaffolds and all models showed declining performance as epistemic demand increased across the four skill categories.Source: §G.2 · p. 59, Figure G.5 · p. 60

G.3 Manual marker annotation

• Among annotated markers, loop instances and wrong reasoning were the most prevalent negative behaviors, while marker patterns also varied by model and scaffold.Claude Sonnet 4.5 produced more explicit reasoning under both scaffolds; GPT-4o showed more nonsense and syntax errors under ReAct, and misunderstood-tool was concentrated in Claude Sonnet 4.5 tool calling.Source: §G.3 · p. 60, Table G.10 · p. 61

G.4 Latent factor modelling

• Two separate 2PL IRT models estimated knowledge and reasoning capabilities for each model–environment pair; the figure summaries show domain variation and unusually high reasoning demands in retrosynthesis.The analysis used binary QA correctness outcomes for both item sets, and the scatter view places retrosynthetic planning far above the knowledge–reasoning diagonal.Source: §G.4 · pp. 61–62, Figure G.6 · p. 62, Figure G.7 · p. 63

G.5 Binomial modeling

• Eight candidate binomial models varied in how they partitioned residual variance, and PSIS-LOO selected the model used for variance decomposition and posterior predictive checks.The candidate family was designed to reflect uncertainty about interaction and grouping assumptions, including whether task difficulty should be modeled by environment or environment-by-scope.Source: §G.5 · pp. 63–64

G.5.1 Model specifications

• Task-level leave-one-out predictions cluster around observed success rates, with R2 > 0.95, supporting the model’s recovery of per-task difficulty beyond in-sample fit.M7 predictions also track observed success rates across the range of empirical success rates.Source: §G.5.1 · p. 67, Figure G.8 · p. 67
• M7 and M3 dominate the other candidate specifications in predictive accuracy and effective-parameter trade-off, and M7 is used in the main text.Only models with ˆR < 1.01 and zero divergences are included in the comparison.Source: Figure G.9 · p. 68
• Total reasoning slopes (ψ + ϕ(e,ℓ)) are positive across all 16 environment × scope cells, ranging from 0.48 to 5.76.Higher reasoning ability predicts better task performance regardless of domain or problem scope.Source: §G.5.1 · pp. 69–70
• In M7, model identity explains 45.2% [41.4, 48.8] of variance, versus 32.5% for environment × scope; scaffold and tool-description verbosity together explain <2%.Task-specific effects contribute 20.7%.Source: §G.5.1 · p. 69, Figure G.11 · p. 70

G.6 Intervention experiments

• The intervention protocol seeds each task with sampled successful or failed trace steps, re-executes tool calls in the live environment, and lets the agent continue autoregressively.Injected prefixes include the first 1 or 2 steps or all but the final 2 or 1 steps.Source: §G.6 · pp. 71–72
• Each task is tested in baseline, success-intervention, and failed-intervention regimes with 15 trials per condition, measuring Pass@k and Pass∧k for k = 1, . . . , 15.A different trace is sampled uniformly for each trial, and all trials use temperature 0.7.Source: §G.6 · p. 72
• Successful traces offer little early benefit in hypothesis-driven inquiry, while late failed-trace injections reduce success to ≤0.12; workflow construction improves gradually and remains ≥0.40 under failed traces.In inquiry environments, successful-trace benefits jump when later steps are injected; in workflow construction, improvement is gradual from early steps onward.Source: Figure G.12 · p. 73

G.7 Token-level log-probability analysis

• The analysis uses mean token-level log-probabilities as an environment-level proxy for model uncertainty, pooling top-1 token log-probabilities across assistant messages and trials.Special control tokens are excluded, and per-environment means are reported.Source: §G.7 · p. 74
• Workflow-construction domains had mean log-probabilities ¯ℓe ≥−0.22, while hypothesis-driven and strategic-reasoning domains shared ¯ℓe = −0.27.Figure G.14 shows consistently lower mean log-probability in hypothesis-driven than workflow-based environments.Source: Figure G.14 · p. 75, §G.7 · p. 75
• Near-complete reasoning trajectories were required to improve performance in the same environments where token-level confidence was lowest.The authors report that this pattern aligns with the gradient in epistemic demand.Source: §G.7 · p. 75

G.8 Verbosity detailed results

• Verbosity has a negligible effect on scores, consistent with the <0.1% variance share attributed to it by the latent factor model.The comparison holds the model, scaffold, environment, and scope fixed.Source: §G.8 · p. 75, Figure G.15 · p. 76
• Workflow and comprehensive score differences relative to brief remain within 0.05 points in every environment.Source: Figure G.15 · p. 76
• Scores stay close across verbosity levels for each model and the two agent scaffolds track each other.Source: Figure G.15 · p. 76

H Reasoning process analysis

• 88% of Claude Sonnet 4.5 and GPT-4o traces showed evidence non-uptake; untested claims appeared in 66% and 62%, respectively.Convergent multi-test evidence and evidence-guided test redesign each appeared in at most 1% of traces; refutation-driven revision appeared in 4% and 1%. The models differed most in fixed-belief traces (47% versus 67%) and stalled revision (21% versus 8%).Source: H Reasoning process analysis · p. 77
• Reasoning breakdowns exceeded productive motifs at every environment level, indicating that undisciplined reasoning was not confined to the highest-difficulty tasks.This is consistent with the paper’s main-text claim that the pattern occurs across task types.Source: H Reasoning process analysis · p. 77

H.1 LLM annotation pipeline

• A two-stage annotation pipeline labels epistemic operations in serialized run histories, then constructs directed edges between related nodes.Claude Sonnet 4.5 performs both stages, using overlapping 20-message windows with a stride of 15 for node labeling and the same windowing for edge construction.Source: §H.1 · p. 77
• Over 80% of validation warnings came from quote mismatches: 56.0% from edge-stage quotes and 25.8% from node-stage quotes.Disallowed edge-node combinations accounted for 11.1% of warnings; no schema violations were observed.Source: §H.1 · p. 78, Table H.13 · p. 79

H.2 Annotations prompts

• Prompt optimization raised human-human PABAK agreement from 85.8% to 92.6% and human-LLM percent agreement from 89.3% to 95.7%.Two experts iteratively refined both annotation stages using a 50-trace training set; gains were similar for node and edge annotations.Source: §H.2 · p. 83, Figure H.16 · p. 84

H.3 Pattern taxonomies

• Productive motifs are graph templates grouped into hypothesis handling, evidence handling, and inquiry control, capturing disciplined inquiry practices such as falsification, triangulation, and test redesign.The templates specify required node types, edge types, and configurations.Source: §H.3 · p. 84, Table H.14 · p. 85
• Reasoning breakdown templates cover untested claims, unrepaired contradictions, and premature commitment; unused or disconnected evidence and unsupported judgments; and fixed beliefs or stalled revision.The taxonomy groups these failures into hypothesis handling, evidence handling, and inquiry control, including failures to revise plans in response to new information.Source: Table H.15 · p. 87

Detailed per-model results

• Across all models, breakdown prevalence exceeds productive-motif prevalence, so the aggregate pattern is not driven by any single system.Table H.16 reports per-model motif and breakdown fractions averaged across environments.Source: §H.4 · p. 88, Table H.16 · p. 88

Environment-level prevalence

• Across every environment and subtask level, reasoning breakdowns outnumber productive motifs, while hypothesis-driven environments have the highest annotation density.The higher density aligns with hypothesis-driven inquiry’s greater demands for iterative belief revision.Source: §H.4 · pp. 88–89, Figure H.17 · p. 89

Scope invariance across subtask levels

• Despite a fourfold increase in task complexity from S1 to S4, neither the prevalence nor the relative ordering of epistemic operations shifts systematically.The reported invariance suggests agents do not adapt their epistemic behavior to broader or more complex problem settings.Source: §H.4 · p. 89, Figure H.18 · p. 90

H.5 Annotation agreement analysis

• Three domain experts independently reviewed 25 of 626 traces across annotation rounds before and after prompt optimization.The analysis reported both human–human and human–LLM agreement, adding PABAK because the skew toward correct labels deflates Cohen’s κ.Source: §H.5 · p. 90
• 92.6% overall human–human agreement and mean PABAK 0.853 indicate substantial concordance; node annotations (PABAK up to 0.976) exceeded edge annotations (0.737–0.913).Overall Cohen’s κ was 0.067 because the correct class was highly prevalent; PABAK adjusts for this imbalance.Source: §H.5 · p. 90, Table H.17 · p. 91
• 95.7% overall human–LLM agreement exceeded the 92.6% human–human baseline.All three reviewers agreed with the LLM on at least 92.8% of node items, while edge agreement ranged from 87.1% to 99.8%.Source: §H.5 · p. 91, Table H.18 · p. 92

H.6 Other agent behavior analysis

• Action profiles are similar across ReAct and tool-calling agents, while models differ more in the tools they invoke across environments.For example, only GPT-4o uses code execution in Resistor; in AFM, Claude and GPT-4o use it more than GPT-Oss.Source: §H.6 · p. 92, Figure H.19 · p. 93
• Verbosity and tool-call patterns vary across environments and models; GPT-Oss is most verbose, while Claude tends to make more calls per trial.Token distributions are similar across agent scaffolds, and both have similar tool-call distributions.Source: §H.6 · p. 93, Figure H.20 · p. 94

H.7 Illustrative traces

• Illustrative traces annotate hypotheses, tests, evidence, judgments, and commitments, marking missing links that should occur in sound reasoning with ×.The excerpts pair each message with extracted epistemic nodes and their connecting edges.Source: §H.7 · p. 98
• In one NMR trace, the agent obtained 20 candidate structures including the correct answer but did not use that list to constrain its guesses.The candidate-list evidence had zero outgoing informs edges, and the agent instead guessed structures one by one.Source: §H.7 · p. 99
• In another NMR trace, the agent interpreted signals as coupled CH2 groups and an acetyl group but moved to mass spectrometry without testing those hypotheses.The structural interpretation had zero outgoing tests edges.Source: §H.7 · p. 100
• In a LAMMPS trace, the agent’s mistaken belief that /potentials/SW was a file persisted alongside a new error hypothesis, with zero hypothesis-update edges.Both beliefs persisted unmodified through the remaining 15 messages despite failures.Source: §H.7 · p. 101
• After a simulated NMR result contradicted its isopropyl-ester hypothesis, the agent dismissed the discrepancy and submitted that structure without revising its belief.The simulation showed a 6H doublet where the experimental data showed 3H; the contradiction had no accompanying updates_to edge.Source: §H.7 · p. 102

Scope of the evaluation

• Each task is an independent episode using ReAct or native tool calling, a flat append-only history, 20–40 calls, and temperature 0.0.The evaluation excludes carry-over learning, advanced orchestration, retrieval, summarisation, and multi-agent architectures; its scores bound what minimal scaffolds achieve, not what is achievable in general. The call budget is fixed across tasks, and temperature sensitivity is not tested.Source: I Scope and limitations · p. 103

Model coverage and compute

• 3 models were evaluated across 138 configuration-environment pairs at five trials each, using approximately 3 billion tokens and an estimated $7.5k in proprietary-model API costs.The models were Claude Sonnet 4.5, GPT-4o, and GPT-OSS (gpt-oss-120b); other prominent model families were omitted. The cost excludes self-hosted infrastructure, interventions, and subtasks, and scaling to more models, trials, or environments carries non-trivial cost.Source: I Scope and limitations · p. 103

Measurement caveats

• 41% of GPT-OSS traces were affected by malformed responses, compared with 6.2% for GPT-4o and 0.5% for Claude Sonnet 4.5.GPT-OSS had 1.26 errors per trace and GPT-4o had 0.28 errors per trial; common failures included invalid JSON in tool arguments and missing XML closing tags. Log-probability analyses were limited to GPT-OSS, and no constrained decoding, output grammars, or retry-with-repair loops were applied, so results reflect systems under the same minimal scaffold rather than model-specific hardening.Source: I Scope and limitations · pp. 103–104