PaperlayerView exact revision on arXiv

Source-linked AI summary

Recording provenance of workflow runs with RO-Crate

Simone Leo, Michael R. Crusoe, Laura Rodríguez-Navas, Raül Sirvent, Alexander Kanitz, Paul De Geest, Rudolf Wittner, Luca Pireddu, Daniel Garijo, José M. Fernández, Iacopo Colonnelli, Matej Gallo, Tazro Ohta, Hirotaka Suetake, Salvador Capella-Gutierrez, Renske de Wit, Bruno P. Kinoshita, Stian Soiland-Reyes

arXiv:2312.07852v2cs.DL

TL;DR

Scientific computation provenance supports traceability, reproducibility, and quality assessment, while existing models have limitations. The paper presents a unified model for workflow-run provenance; several systems implement its profiles, and executions from heterogeneous systems have been compared.

  • Problem

    Recording scientific computation provenance supports traceability, reproducibility, and quality assessment, while existing models have limitations.

    Source: Introduction · p. 2

  • Method

    The paper presents unified RO-Crate profiles for describing prospective and retrospective provenance of computational workflow execution.

    Source: Discussion · p. 24

  • Results

    Several workflow systems have integrated a WRROC profile, and workflow executions on heterogeneous systems have been compared.

    Source: Discussion · p. 24

  • Takeaways & Limitations

    The profiles’ application has been demonstrated through comparisons of workflow executions on heterogeneous systems.

    Source: Discussion · p. 24

    The StreamFlow implementation lacks support for distributed provenance.

    Source: §3.4 · p. 15

Abstract

from arXiv · show

Recording the provenance of scientific computation results is key to the support of traceability, reproducibility and quality assessment of data products. Several data models have been explored to address this need, providing representations of workflow plans and their executions as well as means of packaging the resulting information for archiving and sharing. However, existing approaches tend to lack interoperable adoption across workflow management systems. In this work we present Workflow Run RO-Crate, an extension of RO-Crate (Research Object Crate) and Schema.org to capture the provenance of the execution of computational workflows at different levels of granularity and bundle together all their associated objects (inputs, outputs, code, etc.). The model is supported by a diverse, open community that runs regular meetings, discussing development, maintenance and adoption aspects. Workflow Run RO-Crate is already implemented by several workflow management systems, allowing interoperable comparisons between workflow runs from heterogeneous systems. We describe the model, its alignment to standards such as W3C PROV, and its implementation in six workflow systems. Finally, we illustrate the application of Workflow Run RO-Crate in two use cases of machine learning in the digital image analysis domain. A corresponding RO-Crate for this article is at https://w3id.org/ro/doi/10.5281/zenodo.10368989

Introduction

The introduction motivates interoperable recording of computational workflow-execution provenance together with associated data and executable resources. It presents Workflow Run RO-Crate’s flexible profiles for machine-actionable packaging of prospective and retrospective provenance, including re-execution, and notes its implementations across six workflow management systems and mapping to W3C PROV-O.

Introduction

  • Provenance supports traceability, reproducibility, accountability, and quality assessment.Source: Introduction · p. 2
  • Existing workflow provenance models remain difficult to interoperate because they differ in granularity and workflow assumptions, often have support from only one WMS, and provide only partial references to data items.Source: Introduction · pp. 2–3
  • CWLProv omits tool definitions and file formats, so consumers must consult a workflow system’s native-language definition to reconstruct full prospective provenance.Its RO may also contain metadata files and PROV serializations in different formats, complicating generation and consumption.Source: Introduction · p. 3
  • CWLProv’s detailed execution recording requires substantial WMS support, can conflict with dynamically determined execution plans, and was supported only by cwltool at the time of writing.The introduction also notes limited support for contextual metadata such as authorship, licence, and creation date.Source: Introduction · p. 3
  • The work focuses on capturing computational workflow-execution provenance, including its data and executable resources.RO-Crate packages research data with metadata and associated resources.Source: Introduction · p. 3
  • WRROC provides profiles for packaging machine-actionable prospective and retrospective workflow-run provenance, independent of workflow language or execution system and supporting re-execution.The profiles are designed to represent both the planned and executed aspects of a run.Source: Introduction · p. 4
  • The model has been implemented in six workflow management systems and mapped to W3C PROV-O using SKOS.Source: Introduction · p. 4
  • Profiles offer different levels of detail with minimal mandatory metadata, while the base profile applies to computational processes even without a formal workflow language.This flexibility is intended to accommodate the diverse workflow-management-system landscape.Source: Introduction · p. 4
  • The Workflow Run RO-Crate community sustains the profiles through regular meetings to discuss extensions, issues, and new developments.Source: Introduction · p. 4

2 The Workflow Run RO-Crate profiles

The Workflow Run RO-Crate profiles were developed through community-driven requirements work to cover different workflow execution scenarios and levels of provenance detail. The resulting profiles provide a unified model, with Process Run Crate describing tools that contribute to a computation.

2 The Workflow Run RO-Crate profiles

  • The profiles are the main outcome of an open working group involving workflow users and developers, researchers, and software engineers.The group focuses on workflow-execution provenance and FAIR approaches for data and software.Source: §2 · pp. 4–5
  • Eleven competency questions captured requirements spanning a broad application scope and different levels of provenance granularity.The questions were developed with participants and informed by reviews of existing models.Source: §2 · p. 5
  • The requirements process included public discussion and validation of the profiles’ competency-question implementations with SPARQL queries.All original issues were closed, and the profiles had five official releases on Zenodo.Source: §2 · p. 5
  • Three profiles were developed because use cases differ in how much computational detail they require and whether a workflow formally coordinates the computation.Some require descriptions of all orchestrated computations; others need a black-box account of overall success and results, or cover data flows across applications without a standard workflow language.Source: §2 · p. 5
  • The Process Run Crate describes one or more software applications contributing to a computation, without requiring coordination by a top-level workflow or script.It is the basis for all profiles in the WRROC collection.Source: §2 · p. 5

2. Workflow Run Crate, to describe a computation orchestrated by a predefined

This section describes RO-Crate profiles for recording computational-run provenance at different levels of granularity, from individual application executions to workflow-level provenance and internal workflow details. It covers their interoperability, human- and machine-readable formats, and additional terms for concepts not expressed by existing Schema.org terms.

2.1 Process Run Crate

  • Process Run Crate represents software applications and their executions, linking each action to its application, inputs, outputs, execution times, and responsible agent.When programs are run manually, connecting one action’s output to another’s input can document an implicit workflow.Source: §2.1 · p. 6, Figure 1 · p. 7

2.2 Workflow Run Crate

  • Workflow Run Crate combines Process Run Crate with Workflow RO-Crate to describe WMS-managed workflows, their formal input and output parameters, and the values used in a run.The workflow is represented as a File, SoftwareSourceCode, and ComputationalWorkflow; individual workflow-step executions are left to Provenance Run Crate.Source: §2.2 · pp. 6–7

2.3 Provenance Run Crate

  • Provenance Run Crate extends Workflow Run Crate with tool executions, intermediate outputs, parameters, and logical steps linked to the tools and their executions.It can also represent the workflow management system’s execution and its configuration file.Source: §2.3 · pp. 8–10
  • Parameter connections record how tools consume other tools’ intermediate outputs and how workflow-level parameters map to tool-level parameters, supporting traceability of output dependencies.The authors add wfrun:ParameterConnection and related terms because the existing RO-Crate context lacks suitable terms for these relationships.Source: §2.3 · p. 10
  • Provenance Run Crate is the most detailed profile, extending Workflow Run Crate, which extends Process Run Crate.Process Run Crate provides the basis for the profile collection.Source: §2.3 · p. 10, Figure 5 · p. 12
  • The three profiles support different levels of provenance detail while keeping the RO-Crates they generate interoperable.This gives workflow management systems a path to begin with basic support and gradually implement more detailed representations, reducing effort and complexity.Source: §2.3 · p. 10

2.4 Profile formats

  • WRROC profiles are provided in human-readable HTML and machine-readable RO-Crate formats.Machine-readable metadata can be retrieved from the same URLs using JSON-LD content negotiation with the Accept:application/ld+json HTTP header.Source: §2.4 · pp. 10, 12
  • The model defines new terms for concepts that existing Schema.org terms could not express, and makes them available in multiple formats through content negotiation.Source: §2.4 · p. 12

Implementations

The implementations apply Workflow Run RO-Crate across a conversion tool and workflow management systems to capture workflow-run provenance and execution context. They demonstrate support for interoperable comparisons and, in some cases, re-execution, while exposing system-specific limitations in capturing workflow structure and metadata.

Implementations

  • Seven implementations—one conversion tool and six workflow management systems—show community support and practical usability of the profiles.The implementations were developed independently by different teams.Source: Implementations · p. 12

3.1 Runcrate

  • Runcrate converts CWLProv execution graphs and CWL workflow files into language-agnostic metadata, and its report enables run comparisons across languages and implementations.The report lists execution start and end times and parameter values.Source: §3.1 · p. 13
  • Runcrate can relaunch CWL workflows from crate inputs and successfully re-executed a digital pathology annotation workflow, though full re-execution depends on workflow language and implementation characteristics.Clear mapping of inputs to parameter slots and containerised computational environments can make reproduction easier.Source: §3.1 · p. 13

3.2 Galaxy

  • Galaxy added Workflow Run Crate export in version 23 as an interoperable alternative to its basic invocation export, bundling workflow definition, invocation metadata, and input and output files.Users can export the crate through the web interface or API.Source: §3.2 · p. 13

3.3 COMPSs

  • COMPSs was extended to generate Workflow Run Crate-conformant provenance capturing actual execution details despite its dynamically created workflows and lack of a prior static definition.The workflow entity references the entry-point application run by COMPSs.Source: §3.3 · p. 14
  • COMPSs uses runtime records to add metadata for all input and output assets, whose identification is not required in the profile because they are determined at runtime.Source: §3.3 · p. 14
  • COMPSs assigns unique run IDs using UUIDs in distributed environments or hostname and job ID on supercomputers.The action description includes system and environment details, and can reference a monitoring tool for job profiling.Source: §3.3 · p. 14
  • A COMPSs crate for BackTrackBB on MareNostrum IV captures execution provenance, source files, the workflow graph, and application profiling, and support is integrated from release 3.2.Source: §3.3 · p. 15

3.4 StreamFlow

  • StreamFlow stores prospective and retrospective provenance in a pluggable database, then converts execution records into RO-Crates that include task-level details.Crate generation is available through the streamflow prov command after a CWL workflow completes.Source: §3.4 · p. 15
  • StreamFlow identifies lack of standard support for distributed provenance—execution locations and topology—as a limitation; its configuration-file workaround is product-specific and requires ad-hoc plugins.Source: §3.4 · p. 15

3.5 WfExS-backend

  • WfExS-backend records prospective provenance during staging, including input and workflow URLs, checksums, and metadata from workflow files, containers, and inputs.Source: §3.5 · p. 16
  • WfExS-backend generates prospective and retrospective crates, can represent multiple executions sharing staged inputs, and supports consuming crates to reproduce documented operations.Source: §3.5 · p. 16

3.6 Sapporo

  • Sapporo supports several workflow engines, including cwltool, Toil, and StreamFlow, and offers bioinformatics features such as output feature-statistics calculation.Source: §3.6 · p. 16
  • Sapporo extends the Workflow Run Crate profile for biological features, while Tonkaz compares generated crates to assess reproducibility and support CI/CD and cross-environment checks.The described feature values include DNA alignment mapping rates from BAM files.Source: §3.6 · pp. 16–17
  • Sapporo, a WES wrapper, records files passed in WES requests rather than parsing workflow definitions or using the wrapped WMS’s actual parameters.Its input and output file fields are not linked to formal parameters, so it cannot connect workflow-depicted inputs and outputs to files actually used or generated.Source: §3.6 · p. 17

3.7 Autosubmit

  • Autosubmit’s earlier archive feature stored prospective and retrospective provenance in a ZIP file without structure to distinguish the two types.The archive could later be used to access provenance data or rerun an experiment.Source: §3.7 · p. 17
  • Autosubmit’s flexibility in combining configuration and delegating execution to other engines complicates reliable provenance tracking, while automated validation and coarse-grained workflow guidance were lacking.The cited coarse-grained systems include Autosubmit, Cylc, and ecFlow.Source: §3.7 · p. 17
  • Autosubmit 4.0.100 can generate Workflow Run RO-Crates with prospective provenance extracted from experiment YAML files and preprocessing parameters.The prospective data includes the multiple YAML files, their unified configuration, and parameters used to replace placeholders in script templates.Source: §3.7 · p. 17
  • Autosubmit’s RO-Crate includes retrospective logs and traces alongside prospective provenance and complies with the Workflow Run Crate profile.The implementation used ro-crate-py to create the archive and add JSON-LD metadata.Source: §3.7 · p. 17
  • Autosubmit’s crate records a Project’s type and location but omits the complete Project, leaving configuration and scripts unavailable to users without appropriate access.Many applications run with Autosubmit cannot be publicly shared without consent.Source: §3.7 · p. 18
  • Autosubmit initializes JSON-LD from configuration and local traces, while users can supply partial metadata to add WRROC-required information missing from the configuration model.Examples include licence, authors, inputs, outputs, and formal parameters.Source: §3.7 · p. 18

4 Exemplary use cases

This section presents two use cases applying Workflow Run RO-Crate to machine-learning analyses of human prostate images, illustrating distinct approaches to recording workflow execution and provenance. One combines execution-focused records with CPM provenance to link workflow steps and locate research artefacts across organisations.

4.1 Provenance Run Crate for digital pathology

  • Crates from cwltool and StreamFlow were very similar and described the computation in essentially the same way, making them fully interoperable.The representation includes four actions: the workflow, two tissue-extraction runs, and a tumour-classification run.Source: §4.1 · p. 19
  • The digital-pathology crates supported automatic re-execution, while exposing that reproducibility depends on workflow characteristics and practices such as containerisation.Handling MIRAX inputs motivated a specification for composite datasets represented as collections of files and directories; parameter connections also reveal how workflow-level inputs map to tool-level parameters.Source: §4.1 · p. 21

4.2 Process Run Crate and CPM RO-Crate for cancer

  • The example highlights interdependencies between pipeline steps, while CPM files add execution details such as dataset splits and training iterations.The RO-Crate also represents a model checkpoint after the second training iteration, linked to the corresponding stored model file.Source: §4.2 · p. 22
  • The example does not include provenance for precursor tissues and biological samples because the publicly available input database lacked that provenance.The CPM can link input-dataset provenance to precursor provenance, but that information was unavailable for this example.Source: §4.2 · p. 24
  • Combining CPM and RO-Crate enables lookup of computation-related research artefacts across heterogeneous organisations through the underlying provenance chain.The CPM linking mechanism can connect pipeline provenance with that of precursors or successors, allowing the chain to be traversed.Source: §4.2 · p. 24

Discussion

The discussion describes Workflow Run RO-Crate’s WMS-agnostic provenance profiles, metadata coverage, and mapping to W3C PROV, including extensions and distinctions relevant to trusted environments and BioCompute Objects. It also covers how the Five Safes profile supports human review and transparent use of sensitive data in Trusted Research Environments.

Discussion

  • The profiles describe prospective and retrospective workflow-execution provenance, contextual metadata, and associated entities at flexible levels of granularity, independently of the workflow management system.Six workflow management systems had integrated WRROC support, and comparisons of executions across heterogeneous systems demonstrated interoperability.Source: Discussion · p. 24
  • WRROC adds terms, constraints, and recommendations that let consumers programmatically retrieve workflow-provenance information using predefined patterns, addressing competency questions the base RO-Crate specification was not designed for.The data can also be treated with standard RDF tools.Source: Discussion · p. 25
  • Deriving Workflow Run Crate and Provenance Run Crate from Workflow RO-Crate makes them compatible with WorkflowHub, enabling runs to reference their workflow and be registered, found, and shared.The inheritance mechanism reuses specifications developed for Workflow RO-Crate, including guidelines for prospective provenance.Source: Discussion · p. 25

5.1 Evaluation of metadata coverage using runcrate convert

  • 13 of 20 provenance subtypes (65%) were represented by WRROC—9 fully and 4 partially—compared with 8 by CWLProv RDF (3 fully and 5 partially).The analysis also found improved representation of workflow parameters (WF2), which were insufficiently described in CWLProv RDF.Source: §5.1 · p. 26
  • runcrate preserves most provenance metadata identified as relevant in realistic RO use cases, with richer representations of workflow tools and the workflow itself than CWLProv.The analysis also reports that the Workflow Run RO-Crate RDF preserves metadata previously identified as relevant.Source: §5.1 · p. 26

5.2 Workflow Run RO-Crate and the W3C PROV standard

  • Provenance Run Crate is the Workflow Run RO-Crate profile that most closely matches CWLProv’s level of detail, and its classes and relationships are mapped to W3C PROV concepts using SKOS.A machine-readable version of the mapping is available in the accompanying RO-Crate [112,113].Source: §5.2 · p. 26, Table 3 · p. 28

5.3 Five Safes Workflow Run Crate

  • The Five Safes profile extends Workflow Run Crate for Trusted Research Environments, supporting human review and transparency about sensitive-data use.It follows the Five Safes Framework and was developed for federated workflow execution across UK TREs.Source: §5.3 · pp. 26–27
  • A workflow-run request crate references a pre-approved workflow and project details for assessment against the TRE’s agreement policy for a sensitive dataset.The crate proceeds through TRE-internal phases including validation, sign-off, workflow execution, and disclosure control; the final crate is safe to publish and conforms to the Workflow Run Crate profile.Source: §5.3 · pp. 26–27

5.4 Biocompute Object RO-Crate

  • IEEE 2791-2020, colloquially known as Biocompute Objects (BCO), primarily conveys a computational method for manual regulatory review and further reuse, with values serving as an exemplar run, while Workflow Run Crate documents a particular workflow execution.Source: §5.4 · pp. 27–28

6 Conclusion and future work

Workflow Run RO-Crate packages prospective and retrospective workflow provenance through profiles that capture increasing levels of detail while keeping required metadata minimal. The authors report adoption in six workflow management systems and application in real use cases.

6 Conclusion and future work

  • The profile collection packages prospective and retrospective provenance for workflow execution, including inputs, outputs, and code.It extends RO-Crate and Schema.org to capture contextual information about execution.Source: §6 · pp. 28–29
  • Process Run, Workflow Run, and Provenance Run Crates capture provenance at increasing levels of detail and complexity.The approach minimizes mandatory metadata items.Source: §6 · p. 29
  • Flexible profiles adapt to diverse workflow management systems and modulate implementation effort according to use-case requirements.The base profile is highlighted as one element of this flexibility.Source: §6 · p. 29
  • Adoption in six workflow management systems and application in real use cases demonstrate uptake of Workflow Run RO-Crate.The cited examples include Galaxy, StreamFlow, and COMPSs.Source: §6 · p. 29
  • The runcrate toolkit provides inspection, conversion, and re-execution functionalities for Workflow Run RO-Crates.The paper describes it as implemented alongside the profile collection.Source: §6 · p. 29
  • The profiles and surrounding software remain under development as new requirements and use cases arise.Planned work includes improving toolkit support for creating and querying crates and implementing automated conformance validation.Source: §6 · p. 29
  • Planned work includes examining WES and TES interoperability and building tools to convert between their requests and RO-Crates.The proposed tools would support re-execution on GA4GH Cloud API-powered infrastructure and bundling information from those APIs into RO-Crates.Source: §6 · p. 29
  • An open community maintains and develops Workflow Run RO-Crate through regular meetings and shared discussion channels.The authors expect community activity to support further adoption across workflow systems and use cases.Source: §6 · p. 29

Related papers

Loading 2312.07852v2…