Source-linked AI summary
Designing Collaborative AI-Driven Workflows for Scientific Software Engineering
Akash Dhruv, Max Knobbe, Pedro Machado, Anshu Dubey
TL;DR
Scientific software changes require scarce domain expertise, while agent-generated code can be difficult to verify. The paper proposes staged workflows with human approval and numerical checks, then compares strategies and models on translating MCFM from Fortran to C++. Across fourteen experiments, an author–reviewer loop matched the multi-agent workflow’s output at roughly one third of its cost per file settled.
Problem
Scientific software changes depend on scarce domain expertise, and agent-generated code is not guaranteed correct and can be difficult to supervise over long tasks.
Source: §1 · p. 1Method
The workflows have domain experts specify and plan stages, agents perform bounded implementation, and numerical comparisons plus human approval gate progression.
Source: §1 · p. 1, §5 · p. 11Results
A single author–reviewer loop matched the multi-agent workflow’s output at roughly one third of the cost per file settled across fourteen versions of the translation task.
Source: §5 · p. 11Takeaways & Limitations
The authors conclude that workflow architecture and execution policy provide the control needed for scientific-software agents, while concurrency without careful design did not improve outcomes and increased cost.
Source: §1 · p. 2, §5 · p. 11A gap in the shared specification left Fortran kind parameters unresolved, and both incompatible C++ representations passed the benchmark because the converted code never referenced them directly.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Agentic artificial intelligence systems can carry out a broad range of tasks in software engineering and scientific research, from writing and translating code to running workflows for data analysis and visualization. In scientific computing, the difficulty is verifying that agent-generated code is both correct and understandable to teams whose members bring different areas of expertise. We therefore argue that these systems are best used within collaborative team structures rather than as full automation. In the workflows we propose, domain experts write the specification and plan, and agents operate under a deterministic orchestration pattern to write the target code. Each stage ends with a numerical comparison against the reference code and requires human review and approval before the next begins. We evaluate these workflows on the translation of a large high-energy physics application from Fortran to C++, running the same task under different orchestrators, design patterns, and models. Across fourteen experiments, a simple author--reviewer loop with enforced limits completed a comparable number of files to a multi-agent workflow at about one-third of the cost per file.
1. Introduction
Scientific software is difficult to maintain because safe changes require scarce scientific expertise, while agent-generated code may be incorrect and hard to supervise. The paper proposes staged, human-approved agent workflows and evaluates them on translating MCFM from Fortran to C++.
1. Introduction
- Scientific software is under-resourced, and its long-lived codebases preserve domain knowledge that is rarely documented, making substantial changes dependent on scarce expert time.Modernization aims to preserve that knowledge and keep legacy code usable.Source: §1 · p. 1
- Agentic systems can lower development costs, but their outputs are not guaranteed correct and unconstrained supervision can increase workload and risk undetected errors.Such errors may accumulate, harming maintainability and numerical correctness.Source: §1 · p. 1
- The proposed workflows divide work into reviewable stages: domain experts specify and plan, agents implement, numerical comparisons check results, and human approval gates progress.The design separates supervision, implementation, and acceptance to keep developers engaged and preserve their understanding of the code.Source: §1 · p. 1
- A computational Lab Notebook provides a controlled, versioned record of code, configuration, and each stage’s agentic layer, enabling tasks to be rerun across orchestrators.It supports consistent record keeping and reproducibility for computational work.Source: §1 · pp. 1–2
- The case study translates MCFM, a Fortran parton-level Monte Carlo program, to C++ for HPC and modularity goals, with existing routines providing numerical references for correctness checks.The paper extends CodeScribe, previously used to generate Fortran–C++ interface layers, to design and evaluate agentic workflows.Source: §1 · p. 2
- The same translation task under different strategies and models showed that adding concurrency without careful workflow design did not improve outcomes and increased cost.The authors compare a simple author–reviewer loop with a multi-agent workflow.Source: §1 · p. 2
2. Collaborative Workflow Architecture
The architecture keeps humans responsible for objectives and acceptance while agents perform bounded work; stages advance only after numerical verification and human approval.
2. Collaborative Workflow Architecture
- Domain experts write specifications and plans, agents produce code, and experts accept it at a verification boundary.Software engineers maintain the tools and workflow; numerical checks accepted by domain experts determine whether a stage is finished.Source: §2 · p. 2, §2.1 · p. 2
2.1. Principles
- Humans set objectives and resolve correctness questions, while agents handle broad, repetitive tasks and each stage requires an accepted numerical check.Compilation and the agent’s own report are not sufficient for acceptance.Source: §2.1 · p. 2
2.2. Lab Notebook
- The Lab Notebook is a shared, versioned directory tree that records approvals and run history outside any single agent or orchestrator.Humans own the specification, plan, and approval files; agents can modify only their log, and transformation state can be paused, resumed, or handed off.Source: §2.2 · pp. 2–3
- In the MCFM notebook, configuration inherits downward, while collaborators keep machine-specific settings in separate subtrees and software and tests follow project practices.The project root holds common shell configuration; separate software and tests directories track the relevant repositories and build practices.Source: §2.2 · p. 2
2.3. Staged work
- Large changes are divided into stages chosen at each review boundary, beginning with testable code and expanding outward for verification against a reference.Small stages can be checked against a concrete reference and assigned by team expertise.Source: §2.3 · pp. 3–4
2.4. Orchestration patterns
- The compared orchestration patterns are an author–reviewer loop and a multi-agent workflow with parallel file authoring and serial integration.The multi-agent workflow routes files by complexity and uses a single integrator for the shared build and verification.Source: §2.4 · pp. 4–5
- CodeScribe cycles an author pass and reviewer pass, carries pending work forward, and enforces tool and iteration limits between model calls.It can stop early when the author reports completion or the reviewer finds no pending items or blockers; acceptance remains outside the loop.Source: §2.4 · p. 4
3. Case Study: MCFM
The case study tests agentic workflows on modernizing the MCFM Fortran codebase, with translation verified against its benchmarks. Fourteen runs vary orchestration pattern and model while holding the starting commit and agentic layer fixed.
3. Case Study: MCFM
- MCFM is a large Fortran program for hadron-collider phenomenology, with hundreds of source files and deep call trees.Source: §3 · pp. 5–6
3.1. Task, target and verification criteria
- Translation first converts each Fortran unit to C++ with bindings for existing Fortran callers, then ports the C++ to device-portable kernels.Source: §3.1 · p. 6
- A translated file is verified only if perturbing it changes a benchmark result and its unperturbed result matches the reference within a fixed tolerance.Source: §3.1 · p. 6
3.2. The Lab Notebook and the Design of Experiments
- The batch transformation tracks file readiness, oracle status, and agent-reported outcomes; a gate limits new groups when approvals are pending or a file has failed.Source: §3.2 · p. 6
- Fourteen runs began from the same MCFM commit and the same 445 candidate files.Source: §3.2 · p. 6
- Pairwise comparisons isolate orchestration pattern or base agent and execution policy, while csloop runs also vary model family.Source: §3.2 · p. 6
- Because agentic runs are stochastic, most configurations were replicated, but ccworkflow Opus 5 and ccloop Sonnet 5 each had only one run.Source: §3.2 · p. 7
4. Evaluation
The evaluation covers fourteen runs of the MCFM translation task, comparing cost, workflow patterns, execution policies, models, and file selection. It finds lower per-file costs for most loop runs than for the multi-agent workflow, alongside differences in enforcement and model behavior.
4. Evaluation
- The transcript shows a blocked awk pipeline prompting a Python fallback; each rejection costs an iteration, while the run completed four of five configured cycles.Source: §4 · p. 7, Figure 6 · p. 8
4.1. Cost Metrics and Measurement
- Settled files were counted from produced git branches, and Figure 7 normalizes cost, time, tool calls, and input tokens by settled file.Table 2’s loop counts differ by orchestrator: csloop and ccloop can end at a fixed budget, completion message, or approval gate; ccworkflow ends only at approval.Source: §4.1 · pp. 7–8, Figure 7 · p. 9
4.2. A loop against a multi-agent workflow
- About 40% as much cost for the same files: ccloop matched ccworkflow’s deliverable in about the same time, while most loop runs used less cost, fewer calls, and fewer tokens per settled file.Source: §4.2 · p. 8, Figure 7 · p. 9
- Parallel authoring did not repay duplicated reading: each independent file was small relative to its shared background material, and integration was serial.Source: §4.2 · p. 9
4.3. A specialized against a general-purpose agent
- One ccloop author phase reached 67 iterations against a 30-iteration limit, while csloop never exceeded its bound or deviated from policy.Claude Code’s private memory also influenced ccloop’s performance; CodeScribe runs were started fresh.Source: §4.3 · p. 9
4.4. Comparison across models
- Sonnet 5 used several times as many tool calls per settled file as every other csloop run and finished fewer files; GPT 5.6-Sol R10–R11 performed similarly per file to Opus runs at similar token cost.Source: §4.4 · p. 10
- Two of three Opus 5 runs chose real C++ types for precision kinds, while GPT 5.6-Sol runs wrote integers; the choice did not affect benchmark results because translated code used type keywords directly.Source: §4.4 · p. 10
4.5. Model decision making versus tooling
- All fourteen runs picked nearly the same files; Doxygen recommended the precision-kinds file at fan-in 8 despite its use in more than 200 files, and the specification left its treatment unclear.CodeScribe translated that file, while Claude Code runs used memory to skip it and, in some runs, the Mods/ directory.Source: §4.5 · p. 10
5. Conclusion
The paper compares collaborative orchestration variants for translating MCFM and argues for workflows that preserve human ownership of intent and acceptance. A single author–reviewer loop matched the multi-agent workflow’s output at roughly one third of the cost per file settled.
5. Conclusion
- Figure 8 tracks when runs first named files in modules they later settled, with colors indicating modules and n/a marking runs that settled nothing.The figure covers fourteen transformations/mcfm-translate runs.Source: Figure 8 · p. 11
- Humans retain ownership of intent and acceptance, while explicit numerical checks and recorded human approval gate progress between stages.The workflow checks that required conditions are confirmed before a file counts as verified.Source: §5 · p. 11
- A single author–reviewer loop matched the multi-agent workflow’s output at roughly one third of the cost per file settled.The comparison covered fourteen versions of the same translation task.Source: §5 · p. 11
- Models chose similar files largely because the orchestrator’s ranking tool determined which files were ready to translate.An unrecorded orchestrator memory note influenced agent decisions in three later runs, illustrating that assumptions can carry across runs.Source: §5 · p. 11
- The specification gap for Fortran kind parameters produced incompatible C++ representations that both passed the benchmark because the constants were never directly referenced.Only the first Claude Code run flagged the gap for human review rather than resolving it.Source: §5 · p. 11
- The conclusion presents shared agentic layers as a way to organize science campaigns and systematic AI use, while noting open questions about model APIs and execution policy.The model API exposes controls for prompt caching and tool use; exposing reasoning summaries also raises security risks.Source: §5 · p. 11
Appendix A. Dataset Archive
The dataset archive documents agent trajectories and code changes from the evaluations and includes scripts to reproduce the study and support further evaluations.
Appendix A. Dataset Archive
- The archive contains transformations and agent-trajectory datasets organized by date, with separate logs for CodeScribe and Claude Code runs.The evaluations used the Lab Notebook archived on Zenodo.Source: Appendix A · p. 12
- Claude Code logs are JSONL and CodeScribe loop-agent logs are TOML; a metadata agent collects run records and creates a GitHub branch for code changes.The logs include tool-call and token-usage records, while reasoning summaries and API-returned hashes are scrubbed.Source: Appendix A · p. 12
- The archive includes scripts that reproduce the study, and the dataset is released under the MIT license with a dataset card.The dataset can also support further evaluations.Source: Appendix A · p. 12
- The reference Lab Notebook omits the converted MCFM code but includes scripts that clone its corresponding repository.The scripts clone the repository under software/mcfm.Source: Appendix A · p. 12
Appendix B. Declaration of Generative AI Use
The authors report using AI tools to support workflow development and data analysis, while stating that they reviewed and verified all AI-generated outputs.
Appendix B. Declaration of Generative AI Use
- The authors used AI tools for workflow scripts, exploring agentic approaches, data cleaning, and statistical analysis, and reviewed and verified all generated outputs.They take responsibility for the accuracy and integrity of the work.Source: Appendix B · p. 12