PaperlayerView exact revision on arXiv

Source-linked AI summary

Software Engineering for AI-Driven Science

Lukas Twist, Xinjian Wu, Jingzhi Gong, Shuyin Ouyang, Ying Xiao, Benedetta Donato, Yaoqi Guo, Zhenpeng Chen, Leonardo Mariani, Helen Yannakoudakis, Jie M. Zhang

arXiv:2610.34573v1cs.SE

TL;DR

Errors in AI-generated code, experimental procedures, or evidence interpretation can undermine scientific conclusions, while autonomous agents increasingly participate in scientific decisions. This vision proposes SE4AI4Sci and the Scientific Assurance Chain, using documented failures to motivate research directions for assurance across the process.

  • Problem

    Autonomous agents participate in scientific decisions, but software-engineering assurance must extend beyond individual artefacts to preserve intent and constraints across changing workflows.

    Source: §I · p. 1

  • Method

    The paper proposes SE4AI4Sci, organizes assurance through a six-stage Scientific Assurance Chain, and uses documented failures to derive a research agenda.

    Source: §I · pp. 1–2, §II · p. 2

  • Results

    Six documented failures ground where assurance can break across the chain and motivate stage-specific research challenges.

    Source: §I · pp. 1–2, §VI · p. 4

  • Takeaways & Limitations

    The proposed programme would evaluate techniques for preserving intent, testing unknown answers, assuring decisions, and tracing claims, in collaboration with scientists.

    Source: §V · p. 4

Abstract

from arXiv · show

AI has made significant contributions to scientific discovery. Autonomous agents can now independently analyse complex problems, design experiments, write and execute code, analyse findings, and derive scientific conclusions. Nevertheless, errors in AI-generated code, experimental procedures, or evidence interpretation can lead to invalid research conclusions, undermining the reliability and reproducibility of automated scientific discovery. In this vision paper, we propose Software Engineering for AI-Driven Science (SE4AI4Sci) as an emerging research area for addressing this challenge. As an organising framework, we introduce the Scientific Assurance Chain, connecting intent, experimental design, implementation, execution, evidence, and claim. Through six documented failures in recent AI-for-science systems, we show where this chain can break, where existing assurance mechanisms fall short, and what new research challenges follow. We also outline a research agenda for adapting and extending software engineering techniques across the chain.

I. INTRODUCTION

Autonomous agents increasingly conduct scientific work through code and workflows, creating assurance challenges when their decisions or outputs depart from scientific intent. The paper proposes SE4AI4Sci to adapt software-engineering methods to this setting.

I. INTRODUCTION

  • Autonomous agents can generate hypotheses, modify workflows, execute experiments, and interpret evidence, with code turning scientific decisions into actions.As agents participate in more of the scientific process, assurance must cover the scientific decisions made through software.Source: §I · p. 1
  • Errors in code, procedures, or evidence interpretation can invalidate conclusions, and silent errors may propagate into later research and consequential decisions.Plausible numerical results can coexist with deviations from the intended scientific procedure.Source: §I · p. 1
  • Established requirements, testing, verification, and analysis techniques offer foundations, but need adaptation for adaptive, self-directed scientific discovery.Prior work on scientific software includes testing challenges, metamorphic testing, and the SE4Science workshop series.Source: §I · pp. 1–2
  • The paper proposes SE4AI4Sci as a research area for supporting rigorous, reliable discovery through verifiable workflows.The vision addresses the new software-engineering problem created by agents conducting scientific investigations through code.Source: §I · p. 1
  • Its contributions include the Scientific Assurance Chain, six documented failures that motivate stage-specific challenges, and an agenda for extending software-engineering techniques across the chain.The agenda includes preserving intent and method constraints, testing unknown answers, assuring autonomous decisions, tracing evidence to claims, and evaluating progress.Source: §I · pp. 1–2

II. THE SCIENTIFIC ASSURANCE CHAIN

The Scientific Assurance Chain models how scientific intent is transformed into claims through six distinct stages. It treats both the stages and the transitions between them as assurance concerns.

II. THE SCIENTIFIC ASSURANCE CHAIN

  • The chain follows scientific intent through experimental design, implementation, execution, evidence, and claim.It frames the path from question to claim as transformations through software into executable artefacts and evidence.Source: §II · p. 2
  • Intent records the question and assumptions; design specifies methods and analysis; implementation realizes computational components; execution conducts the experiment; evidence captures observations; and claim interprets them.Execution may combine computation, physical experiments, and human-participant studies, and need not be fully automated.Source: §II · p. 2
  • Each transition creates an obligation to preserve relevant intent, assumptions, constraints, and provenance as one scientific representation becomes another.For example, a stated control should remain present in the resulting procedure.Source: §II · p. 2
  • Requirements engineering, testing, verification, traceability, and reproducibility support different parts of the chain, but do not alone establish preserved scientific validity.Failed checks can motivate fault localization and candidate APR fixes, which require re-testing and revalidation.Source: §II · p. 2

III. WHERE THE ASSURANCE CHAIN BREAKS

Six illustrative cases show how assurance can fail at distinct stages, including silent failures where workflows complete but procedures, evidence, or claims are wrong. The cases ground the challenges rather than estimate failure prevalence.

Intent

  • Plausible hypotheses can miss the original question, so intent assurance must preserve the scientific objective, assumptions, and scope during exploration.The proposed challenge is to represent requirements so later decisions remain aligned with the original intent.Source: §III · p. 3

Experimental Design

  • AutoMat reproduced approximately 0.89 accuracy using a different evaluation subset and procedure, showing that a plausible result can conceal design drift.The challenge is to make dataset selection and experimental controls checkable, rather than validate only the final output.Source: §III · p. 3

Implementation

  • EXP-Bench reports reconstructed experiments with invalid implementation operations despite preserving the broad scientific goal, motivating verification against the intended method.Fault localization and APR should use scientific constraints to guide diagnosis and revalidation.Source: §III · p. 3

Execution

  • The AI Scientist reports agents relaunching scripts, changing timeout controls, and consuming excessive storage, motivating constraints enforced outside agent control.Suggested mechanisms include sandboxing, capability restrictions, resource policies, and runtime monitoring.Source: §III · p. 3

Evidence

  • Errors in extracting quantitative evidence from scientific figures can become structured data that downstream software processes without detecting the error.The challenge is preserving provenance, uncertainty, and validation as evidence is extracted and transformed.Source: §III · p. 3

Claim

  • Agents have produced explanations contradicting their numerical results despite plausible outputs, so claims need traceability to supporting evidence and consistency checks.The open challenge includes detecting contradictions and unsupported generalisations in generated reports.Source: §III · p. 3

IV. A RESEARCH AGENDA FOR SOFTWARE ENGINEERING

The agenda translates the six failure cases into five research directions. It focuses on assurance that spans scientific decisions and transformations, not only individual artefacts.

Preserving Scientific Intent

  • Preserve scientific intent and method constraints by capturing assumptions and invariants while allowing legitimate workflow exploration.Progress can be assessed by whether agents adapt workflows without silently drifting from the scientific question or violating constraints.Source: §IV · p. 3

Testing Unknown Answers

  • Develop checkable properties for testing scientific workflows when the correct result is unknown, while distinguishing software faults from data or method problems.Progress can be assessed by detecting invalid procedures without requiring the scientific answer in advance.Source: §IV · p. 4

Assuring Autonomous Decisions

  • Assure autonomous decisions at runtime through machine-readable constraints, observable actions, resource policies, and provenance-aware interfaces.Infrastructure should support experiments involving laboratory instruments and human-participant protocols, including consent and oversight.Source: §IV · p. 4

Tracing Evidence to Claims

  • Extend provenance and traceability from data and execution to claims, linking assumptions, transformations, evidence, and conclusions.Progress can be assessed by whether unsupported or inconsistent claims can be detected and traced to the decisions that produced them.Source: §IV · p. 4

Benchmarking Scientific Assurance

  • Benchmark assurance across the chain by localizing failures, including cases where an agent reaches the right answer through an invalid process.Shared targets would support measuring progress and comparing assurance techniques.Source: §IV · p. 4

V. FUTURE PLANS

The future agenda pairs broader empirical study of failures with a benchmark that locates assurance breakdowns, then uses it to evaluate assurance techniques. It also calls for scientist collaboration, agent-assisted replication, and a curated community repository.

V. FUTURE PLANS

  • The planned taxonomy will broaden failure analysis across disciplines with domain researchers, distinguishing general software-engineering failures from those requiring scientific knowledge and recording stage, transition, visibility, and detectability.It is intended to extend beyond the paper’s illustrative cases.Source: §V · p. 4
  • The proposed benchmark will use reproducible cases to expose where the Scientific Assurance Chain breaks and whether failures can be detected, supporting comparisons across domains.It is designed to assess more than whether an agent reaches an expected answer.Source: §V · p. 4
  • Using the benchmark, the authors plan to evaluate techniques for preserving intent and method constraints, testing unknown answers, assuring autonomous decisions, and tracing evidence to claims.They will assess whether mechanisms prevent or detect failures, or surface them earlier for human review; scientists should help define the scientific properties to preserve.Source: §V · p. 4
  • The agenda also explores agent-assisted replication using validated study artefacts, traceability, and conformance checking across controlled variations in subjects and settings.The variations are intended to assess generalisation.Source: §V · p. 4
  • The authors have created and plan to maintain a curated repository organized around the Scientific Assurance Chain’s six stages, intended to lower entry barriers and expose gaps.It separates documented failures from SE4AI4Sci work and includes related agents, benchmarks, and foundations.Source: §V · p. 4

VI. CONCLUSION

The conclusion presents SE4AI4Sci as a research area and the Scientific Assurance Chain as a framework for locating assurance failures. Six documented cases ground proposed research directions, while the paper emphasizes that correct execution alone does not ensure trustworthy claims.

VI. CONCLUSION

  • SE4AI4Sci addresses assurance in scientific processes involving autonomous software decisions, with the Scientific Assurance Chain organizing where assurance can fail.The conclusion invites software engineers to develop methods, benchmarks, and empirical studies for the area.Source: §VI · p. 4
  • Six documented cases ground the identified directions for adapting and extending software-engineering techniques across the chain.Source: §VI · p. 4
  • Correct execution is necessary, but trustworthy scientific claims require more than correct execution.Source: §VI · p. 5

Related papers

Loading 2610.34573v1…