OpenAI4S: Code as Action, Science as Sessions

1 Introduction

• Long-running research needs persistent computational state and explicit provenance so researchers can inspect how results were produced, resume work, and assess reproducibility.Intermediate trajectories, fitted models, and analysis objects can be expensive to reconstruct; failures and restarts require artifact management and recovery mechanisms.Source: §1 · pp. 1–2
• OpenAI4S separates structured orchestration from complete code-cell execution in persistent Python and R kernels, with Python cells able to invoke audited host services.Its append-only Action Ledger and per-cell records link actions to outcomes; artifact versions, environment records, and checkpoints support inspection, branching, and bounded recovery.Source: §1 · pp. 2–3
• 7.83 overall score versus 5.7–6.4 for a general-purpose coding harness across 36 scenarios in six tasks, with largest gains on long, computation-intensive workflows.The largest gains were on molecular dynamics and protein binder design.Source: §1 · pp. 2–3
• Configurable sandboxing, permission mediation, code screening, and trajectory screening provide complementary safeguards for selected biological and chemical risks.The evaluation protocol scores repository-level task accuracy, workflow completeness, and reproducibility through deterministic verification followed by constrained semantic auditing.Source: §1 · p. 3

Agents for scientific discovery

• Discovery agents span lab automation [1, 2, 15], biomedical tools [3, 4], and research-report production [7, 8]; OpenAI4S targets session history, artifact records, and recovery.The paper presents OpenAI4S as a layer to use alongside domain tools and research-planning methods.Source: §2 · p. 3

Coding harnesses and code as action

• Code-first frameworks establish executable actions and persistent state [10–13]; OpenAI4S combines Python/R kernels, audited host calls, structured orchestration, and research-session records.TaskWeaver retains session state and includes code verification, execution logs, and isolation mechanisms [11].Source: §2 · p. 3

Reproducibility and provenance in computational science

• Computational notebooks and FAIR principles motivate preserving research records [19–21], but captured provenance and package integrity alone do not guarantee reproducible results.Input lineage depends on supported tracking mechanisms and configuration; static variable dependencies are conservative estimates.Source: §2 · p. 3

Evaluating scientific agents

• Many scientific-agent benchmarks score final outputs [22–27], while CORE-bench tests computational reproduction [28]; this protocol assesses repository accuracy, workflow completeness, and reproducibility.It uses binary criteria, gives deterministic checks precedence over an LLM judge, and limits the judge to structured repository evidence.Source: §2 · p. 4

3.1 Overview: from tool-calling agents to a scientific computing runtime

• Complete Python and R code cells run in persistent kernels, connecting scientific computation to session records and recovery mechanisms.The design uses provider-native JSON control calls for orchestration and a scientific computing runtime for code execution.Source: §3.1 · p. 4

3.2 Dual-plane execution architecture

• Python and R kernels retain their own namespaces while alive, and complete cells can combine scientific operations using resident objects.Background jobs use independent Python workers, so their inputs must be prepared explicitly rather than inherited from the foreground namespace.Source: §3.2 · p. 4
• The shared loop routes control calls and code cells through separate channels, while Python cells can synchronously invoke audited host services.Both channels use a common dispatcher for permission checks, screening, and audit; R has no host RPC interface.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 Reproducible research workflow management

• Research sessions connect objectives, agent turns, cells, outputs, and findings; action and cell records preserve execution details, while artifact versions and available lineage add production context.Lineage coverage depends on observed supported accesses and does not guarantee complete dependency attribution for every reported number.Source: §3.3 · pp. 5–6, Figure 2 · p. 5
• Checkpoints and session packages preserve captured workspace state and records, while verified replay can recover eligible computing state but may report incomplete coverage.Packages preserve captured bytes but do not reconstruct arbitrary external services or a new machine’s scientific environment; completion checks do not establish scientific truth.Source: §3.3 · pp. 6–7

3.4 Structural and data safety

• Safety combines kernel sandboxing, permission mediation, and code and trajectory screening, but classifier and screening failure behavior limits what these controls guarantee.The biological and chemical screening prompt is not automatic identity verification, and its stated scope is not comprehensive for all sensitive drugs or synthesis intermediates.Source: §3.4 · pp. 7–8

3.5 Skills

• Skills provide on-demand code recipes and optional kernel helpers for scientific workflows, with 604 bundled recipes spanning curated and bioSkills collections.Execution may use local kernels, configured MCP or model backends, or remote compute, depending on each recipe’s requirements.Source: §3.5 · p. 8

3.6 Implementation

• A provider-neutral engine serves the CLI and web workbench, with SQLite session storage and optional extensions for Notebook execution, lineage, and review.The repository includes over 6,000 offline tests and is released under the MIT license.Source: §3.6 · p. 8

4.1 Science domain

• The 36 scenarios span six scientific tasks, and controlled variations test whether agents adapt shared workflows to changed objectives, inputs, methods, constraints, budgets, or decision criteria.Paired scenario-level comparisons cover 31 scenarios across five tasks; all six tasks contribute to task-level results.Source: §4.1 · p. 8, Table 2 · p. 9

4.2 Quality evaluation

• The evaluator scores scientific task accuracy, workflow completeness, and reproducibility using fine-grained binary criteria and weights of 0.4, 0.35, and 0.25, respectively.Deterministic checks handle verifiable properties, while constrained LLM auditing addresses criteria requiring semantic judgment.Source: §4.2 · pp. 10–11
• A criterion fails if any of its signals fail, while scenario-inapplicable criteria are excluded; hard failures can cap the overall score at 2.0 by default.Hard failures include violations that undermine scientific credibility, such as using withheld evaluation data.Source: §4.2 · p. 12
• The comparison evaluates complete system configurations, so its score differences cannot be attributed to any one mechanism without separate controlled experiments.The configurations combine models, harnesses, prompts, tools, scientific resources, and runtime behavior.Source: §4.2 · p. 12

A Six-task performance

• 7.83 is OpenAI4S’s macro score across six tasks, versus 5.79 for Kimi-K3, 6.36 for GLM-5.2, 6.12 for Opus-4.8, and 8.70 for the GT reference workflow.The comparison uses complete systems under a repository-level evaluator, not matched architecture ablations.Source: Table 3 · p. 12, §4.3 · p. 12

B Scenario-score distributions

• The 31 paired scenarios show stronger central performance and a stronger lower tail for OpenAI4S, with 6.5% scoring below 5, versus 61.3%, 45.2%, and 51.6% for the three baselines.The survival curves remain separated at every score threshold, rather than the aggregate advantage depending on a few outlier scenarios.Source: §4.4 · p. 12, B Scenario-score distributions · p. 13

4.4 Distributional robustness and scientific delivery

• OpenAI4S produced both a final-artifact manifest and a final report in all 15 paired MD and Protein Binder Design scenarios, versus 0/15, 3/15, and 8/15 for the baselines.In MD, it produced reports in 9/9 scenarios by separating final delivery from expensive upstream computation, including for partial or interrupted runs.Source: §4.4 · p. 12

D Empirical survival of scenario scores

• In the protonation-state analysis, 95% of sampled report numbers were traceable to machine-readable evidence.Source: D Empirical survival of scenario scores · p. 13

4.5 Representative workflow failure modes

• The representative cases span syntax errors, truncated execution, unrun modules, substituted methods, and incomplete packaging.The catalyst-screening case shows that OpenAI4S can lose to all three comparison runs when its final package omits required deliverables despite completing the numerical screen.Source: §4.5 · p. 13
• The failures occur at multiple stages between workflow construction and final delivery, rather than only during computation.Source: §4.5 · p. 13

D Empirical survival of scenario scores

• OpenAI4S scores higher in aggregate, has a higher lower-tail floor, and more reliably turns workflow logic into inspectable artifacts, reports and trajectories.The paired-subset survival curves compare the fraction of scenarios at or above each score threshold.Source: §4.5 · p. 13, Figure 5 · p. 13, D Empirical survival of scenario scores · p. 13

5 Discussion

• The complete OpenAI4S system reaches the highest six-task macro score and strongest lower tail on the paired subset, with separation concentrated in inspectable scientific outputs.The discussion highlights long-horizon protein workflows and identifies retrosynthesis as the smallest-effect task.Source: §5 · p. 13

A Dimension-level score differences

• Against Opus-4.8, the raw 0–1 deltas were 0.414 in Accuracy, 0.102 in Workflow and 0.052 in Reproducibility.Source: A Dimension-level score differences · p. 14

D Reporting code does not guarantee a delivered report

• Figure 6 compares dimension-score differences, paired counts for criteria with the largest advantages, and final-artifact and report delivery across 15 MD and Protein Binder Design scenarios.Its final panel contrasts reporting-module presence with actual report delivery in nine MD scenarios.Source: Figure 6 · p. 14
• The comparison uses complete systems with unmatched models, harnesses, prompts, tools and compute, so it does not identify the causal contribution of an individual runtime component.Source: D Reporting code does not guarantee a… · p. 14
• The stated next steps include completing the six-task cross-system scenario matrix and making session packages portable and verifiable across machines.Source: D Reporting code does not guarantee a… · p. 14

Representative workflow failure modes

• The budget-constrained catalyst-screening case shows that OpenAI4S can score below all three comparison runs when its final package omits required report elements despite completing the numerical screen.The omitted items were the required report name, figure, and dependency declaration.Source: D Reporting code does not guarantee a… · p. 14, Figure 7 · p. 15

• The six-task scenario definitions and corresponding LLM-judge evaluation pipeline are publicly available at SciCodeBench-OpenAI4S.Source: Data Availability · p. 15
• The OpenAI4S repository also provides task-specific scenario definitions and implementation details.Source: Data Availability · p. 15

Code Availability

• OpenAI4S is available under the MIT license, with PyPI and desktop distribution paths.The implementation baseline in Section 3.6 is distinct from the version used in benchmark runs.Source: Code Availability · p. 15

Supplementary implementation notes

• The implementation baseline described in Section 3.6 is distinct from the version and configuration used in the reported experiments.The checked baseline is commit 4e96b25 dated 8 September 2026; these implementation notes report no experimental results.Source: Supplementary implementation notes · p. 18

S1. Execution channels and state lifetime

• One outer loop selects native control calls, finalization, or a complete Python/R cell; worker namespaces persist only while their workers remain alive.Background jobs use independent Python workers and require prepared inputs; R cells do not emit completion signals, and data exchange uses workspace files.Source: Supplementary implementation notes · p. 18

S2. What execution records measure

• Cell records associate source, output, errors, and worker identity, but recorded fields do not show that every effect of the code was observed.Resource measures, static dependency estimates, and supported Python object-lineage tracking have coverage and uncertainty boundaries; missing lineage edges do not establish that an output had no inputs.Source: Supplementary implementation notes · p. 18

S3. Defaults, optional coverage and failure behavior

• Safeguards have configurable boundaries: sandbox auto mode may continue in degraded status, screening can allow unmatched risks, and several review features default to off.A failed or unconfigured biosecurity model call allows execution, while ESCALATE is advisory in current CLI/Web paths; completion checks do not verify scientific conclusions.Source: Supplementary implementation notes · p. 19

S4. Persistence, recovery and exchange

• Workspace files and artifact versions can survive worker restarts, but arbitrary in-memory Python and R objects do not.Checkpoints capture content-addressed workspace state and supported session metadata; failed best-effort captures leave no usable recovery point, and session deletion removes its append-only history.Source: Supplementary implementation notes · p. 19
• Recovery validates replay safety, namespace coverage, artifact hashes, and environment identity before publishing a candidate worker, and reports partial or failed outcomes when verification fails.Python skill-load observations cannot independently attest which sidecar executed; failed sidecar capture prevents automatic recovery of that generation.Source: Supplementary implementation notes · p. 19
• Environment manifests and session bundles do not guarantee recreation of another machine’s environments, external services, credentials, hardware, or arbitrary namespace objects.Imported content is quarantined before activation; environment generations support build, verification, activation, and rollback.Source: Supplementary implementation notes · p. 19

S5. Benchmark identifiers used in the main text

• The supplementary notes map representative scenarios to repository identifiers and name evaluator criteria, whose full definitions and weights are distributed with the evaluation pipeline.Examples include lysozyme protonation-state molecular dynamics (md-07), Raman instrument mismatch (1-instrument-mismatch), and budget-constrained hierarchical catalyst screening (scenario-5).Source: Supplementary implementation notes · pp. 19–20