Source-linked AI summary
Methods Included: Standardizing Computational Reuse and Portability with the Common Workflow Language
Michael R. Crusoe, Sanne Abeln, Alexandru Iosup, Peter Amstutz, John Chilton, Nebojša Tijanić, Hervé Ménager, Stian Soiland-Reyes, Bogdan Gavrilovic, Carole Goble
TL;DR
Competing workflow systems and implicit data flows make computational workflows difficult to share and port. CWL offers open, declarative standards for command-line workflows, and the paper reports use across science and engineering alongside production-grade implementations and shared workflow descriptions.
Problem
Competing workflow systems and abstractions that do not explicitly specify data flows limit portability and increase third-party reuse and porting costs.
Source: §1 · p. 2Method
CWL provides a common declarative abstraction for describing command-line tools and workflows, with specified inputs, outputs, and tool connections.
Source: §1 · p. 2Results
Takeaways & Limitations
CWL descriptions support portability and reuse by making tool environments and container use explicit.
Source: §1 · p. 2CWL is designed for command-line-tool-based data analysis and does not cover advanced control flow beyond conditional steps or runtime workflow-graph manipulation.
Source: §3.2 · pp. 7–8
Abstract
from arXiv · showhide
Computational Workflows are widely used in data analysis, enabling innovation and decision-making. In many domains (bioinformatics, image analysis, & radio astronomy) the analysis components are numerous and written in multiple different computer languages by third parties. However, many competing workflow systems exist, severely limiting portability of such workflows, thereby hindering the transfer of workflows between different systems, between different projects and different settings, leading to vendor lock-ins and limiting their generic re-usability. Here we present the Common Workflow Language (CWL) project which produces free and open standards for describing command-line tool based workflows. The CWL standards provide a common but reduced set of abstractions that are both used in practice and implemented in many popular workflow systems. The CWL language is declarative, which allows expressing computational workflows constructed from diverse software tools, executed each through their command-line interface. Being explicit about the runtime environment and any use of software containers enables portability and reuse. Workflows written according to the CWL standards are a reusable description of that analysis that are runnable on a diverse set of computing environments. These descriptions contain enough information for advanced optimization without additional input from workflow authors. The CWL standards support polylingual workflows, enabling portability and reuse of such workflows, easing for example scholarly publication, fulfilling regulatory requirements, collaboration in/between academic research and industry, while reducing implementation costs. CWL has been taken up by a wide variety of domains, and industries and support has been implemented in many major workflow systems.
1 INTRODUCTION
The introduction presents CWL as a common, declarative standard for describing command-line workflows, addressing the difficulty of sharing polylingual workflows across incompatible systems. It describes the project’s open, community-led approach and CWL’s use across scientific and engineering domains.
1 INTRODUCTION
- CWL provides a common abstraction, implemented or implementable in many workflow systems, for communities that need polylingual and multi-party workflows.The authors frame it as a pragmatic set of standards that covers most features used in practice.Source: §1 · p. 2
- A declarative CWL description specifies command-line tools, their inputs and outputs, and how tool outputs can feed later tools.The standards focus on commonly used operations and separate workflow description from execution.Source: §1 · pp. 2–3
- CWL is used across science and engineering, including hydrology, radio astronomy, geospatial analysis, high-energy physics, genomics, and cancer research.The standards are featured in the US FDA-sponsored IEEE Std 2791-2020 and the Netherlands’ National Plan for Open Science; the project also describes a COVID-19 database and analysis resource collaboration.Source: §1 · pp. 2–3
- Figure 1 illustrates a CWL workflow matching genomic sequences to sequence models, dispatching work to four sub-workflows, and collating unique hits.Its command-line steps use different programming languages, and the task connections imply partial ordering and possible concurrency [26].Source: Figure 1 · p. 3
- CWL makes runtime environments and container use explicit, supporting portability and reuse across workflows and computing settings.The standards also support automation, scalability, abstraction, and provenance.Source: §1 · pp. 2–3
- The CWL project develops free, open standards through community collaboration and shared decision-making, with contributions from a diverse community working with industry.The standards are available as open standards and are supported by a Free/Open Source Software ecosystem.Source: §1 · p. 3
2 BACKGROUND ON WORKFLOWS AND STANDARDS FOR WORKFLOWS
Workflows structure computational processes as connected tasks and data flows, helping manage complexity, scale execution, automate tasks, and track provenance. The section explains why shared workflow standards matter and contrasts language-specific approaches with command-line coordination for polylingual workflows.
2.1 Why Workflows?
- Polylingual, multi-party workflows are common in data-intensive fields, but groups often cannot share or collaborate across institutions and infrastructure without costly manual translation.As processes grow complex, undocumented dependencies and informal logs can make scaling, reuse, and understanding past runs difficult.Source: §2.1 · pp. 3–4
- Workflow techniques provide abstraction, scaling, automation, and provenance by structuring components, dependencies, inputs, outputs, and execution records.This structure can support parallel or distributed execution and later queries about tasks, executors, inputs, outputs, and logs.Source: §2.1 · p. 4
2.2 Why Workflow Standards?
- Workflow standards address incompatible systems, vendor and hardware lock-in, and incomplete method descriptions that hinder sharing, reuse, replication, and technology transfer.Structured, machine-readable descriptions can provide needed information for sharing, while workflow-management systems can automate operational details.Source: §2.2 · p. 4
2.3 Sidebar A: Monolingual and Polylingual workflow systems
- Language-specific libraries make workflow structure explicit within a programming language, while polylingual approaches can use add-in libraries or the POSIX command-line interface.CWL uses the POSIX CLI model: string arguments and environment variables coordinate components without requiring additional implementation in each language, at the cost of process startup time and a simple data model.Source: §2.3 · pp. 4–5
3 FEATURES OF THE COMMON WORKFLOW LANGUAGE STANDARDS
CWL specifies command-line tools and composed workflows through explicit descriptions of inputs, outputs, runtime environments, and data flow. The standards target reusable, portable execution, while defining a command-line workflow scope with specific exclusions.
3 FEATURES OF THE COMMON WORKFLOW LANGUAGE STANDARDS
- CWL defines command-line tool descriptions and workflow-composition descriptions, specifying tool inputs, parameter types, invocation flags, and output locations.Descriptions can add documentation, file-format identifiers, container recommendations, and resource hints or requirements.Source: §3 · p. 5, Figure 2 · p. 6
- Explicit runtime environments and data locations, with optional software containers, support execution across local, batch, and cloud environments and reuse of tool definitions.The same workflow description can run across these environments; individual tool definitions can also be reused in new workflows.Source: §3 · pp. 5, 7, Figure 3 · p. 6
3.1 Execution of workflows in CWL format
- Extensions that are optional hints may be ignored by compatible engines, while required extensions must be recognized and may be rejected when an engine cannot execute them.Execution timing and failure recovery are left to workflow engines, and CWL does not define execution APIs or real-time monitoring.Source: §3 · p. 6, §3.1 · p. 7
3 FEATURES OF THE COMMON WORKFLOW LANGUAGE STANDARDS
- Workflow descriptions connect named step inputs and outputs into explicit data flows, supporting parallel execution, scatter repetition, and conditional step skipping.Scatter and conditional execution do not require modifying the underlying tool description.Source: §3 · p. 7
3.1 Execution of workflows in CWL format
- CWL implementations provide streaming, data-aware step placement, and reuse of previously computed step results without requiring workflow rewrites.Result reuse can span workflows when inputs are identical and is controlled by the WorkReuse directive.Source: §3.1 · p. 7
- CWL users and vendors report analyses of 5,000 whole-genome sequences in one execution and a successful 8,000-wide step within a 25,000-container-execution workflow; scalability bottlenecks are hardware.The standards impose no technical limits on file sizes or parallel task counts.Source: §3.1 · p. 7
3.2 When is CWL not useful?
- CWL is out of scope for stateful service interactions, real-time communication between steps, runtime graph edits, advanced control flow, and workflows containing cycles.The described scope is command-line execution with declared file inputs and outputs; the listed exclusions include repeat-until and fixed-count cycles.Source: §3.2 · pp. 7–8
4 OPEN-SOURCE, OPEN STANDARDS, OPEN COMMUNITY
CWL combines code, open standards, and community development to serve diverse stakeholders and foster an active ecosystem. Its principles and supporting tools have accompanied broad implementation and use across workflow systems.
4 OPEN-SOURCE, OPEN STANDARDS, OPEN COMMUNITY
- The project combines code, standards, and community to support diverse stakeholders, fostering free and open-source software and an active ecosystem.Source: §4 · p. 8
4.1 The CWL Principles
- The five principles center the community, remove participation costs, license standards for reuse, let the community set project priorities, and require frequent testing.Free participation and licensing have enabled many implementations; conformance tests assess the reference implementation and help adopters and production developers check correctness.Source: §4.1 · p. 8
4.2.1 Free and Open Source implementations of the CWL standards.
- By 2021, CWL was widely supported in practice, with Galaxy and Pegasus also developing support.Implementations self-certify support for particular technology configurations; the community does not test or certify implementations or stacks.Source: §4.2.1 · p. 8
4.2.2 F/OSS tools and libraries for working with CWL format documents. 27
- The ecosystem includes editor plugins, document-generation tools, multilingual libraries, and Python and Java SDKs generated from the CWL schema.The SDKs let programmers load, modify, and save CWL documents using an object-oriented model corresponding to the standards.Source: §4.2.2 · p. 9, §4.3 · p. 9
4.3 The CWL Ecosystem
- CWLProv demonstrates combining existing standards to represent workflow-execution provenance, though it has so far been implemented only in the reference runner.Source: §4.3 · p. 9
5 CONCLUSION
The conclusion presents CWL as a community-driven response to scattered efforts to define and port workflows. It argues that its reduced, declarative, open standards support reuse and flexibility, with implementations and adoption across communities.
5 CONCLUSION
- CWL is a family of standards for describing command-line tools and workflows made from them, including containers, resource requirements, and workflow-level conditional branching.Source: §5 · p. 9
- CWL offers a common, reduced set of capabilities used in practice and implemented in many popular workflow systems.Source: §5 · p. 9
- Its declarative syntax lets users specify workflows without knowing where they will later run.Source: §5 · p. 9
- The project provides free and open standards rather than a workflow product, runner, or vendor, allowing users, operators, and vendors to avoid lock-in and retain flexibility.Source: §5 · p. 9
- CWL standards are already implemented and adopted, with many production-grade open-source implementations and workflow and tool descriptions available at no cost.Source: §5 · p. 9