Source-linked AI summary
Parsl: Pervasive Parallel Programming in Python
Yadu Babuji, Anna Woodard, Zhuozhao Li, Daniel S. Katz, Ben Clifford, Rohan Kumar, Lukasz Lacinski, Ryan Chard, Justin M. Wozniak, Ian Foster, Michael Wilde, Kyle Chard
TL;DR
Python-based scientific workflows need intuitive ways to assemble components and exploit parallel hardware as data grows and sequential processing plateaus. Parsl extends Python with decorators, futures, dynamic dependency graphs, and configurable executors; its evaluations report low overhead, high throughput, and extreme-scale execution across diverse workloads.
Problem
Growing component-based software and increasing data sizes require parallelism to be expressed naturally in high-level Python programs.
Method
Parsl augments Python with decorators, futures, dynamic dependency graphs, and extensible executors for configurable parallel execution.
Results
Parsl fills varied parallel-Python use cases with execution overhead below 5 ms, scalability to approximately 8000 nodes, and throughput of approximately 1200 tasks per second.
Takeaways & Limitations
Parsl supports many-task, interactive, online, and machine-learning applications across fields including biology, cosmology, and materials science.
Takeaways & Limitations
At extreme scale, machine and node failures can cause loss of an entire MPI application, so the authors recommend several smaller MPI worker pools within one scheduler job.
Abstract
from arXiv · showhide
High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs. Here, we present Parsl, a parallel scripting library that augments Python with simple, scalable, and flexible constructs for encoding parallelism. These constructs allow Parsl to construct a dynamic dependency graph of components that it can then execute efficiently on one or many processors. Parsl is designed for scalability, with an extensible set of executors tailored to different use cases, such as low-latency, high-throughput, or extreme-scale execution. We show, via experiments on the Blue Waters supercomputer, that Parsl executors can allow Python scripts to execute components with as little as 5 ms of overhead, scale to more than 250 000 workers across more than 8000 nodes, and process upward of 1200 tasks per second. Other Parsl features simplify the construction and execution of composite programs by supporting elastic provisioning and scaling of infrastructure, fault-tolerant execution, and integrated wide-area data management. We show that these capabilities satisfy the needs of many-task, interactive, online, and machine learning applications in fields such as biology, cosmology, and materials science.
1 INTRODUCTION
Parsl extends Python with decorators and futures for expressing compositional parallelism while separating program logic from execution configuration. Its extensible executors support scalable execution across diverse architectures and workloads.
- Contribution: Parsl uses Python decorators to express parallelism while composing Python functions and components written in other languages.This supports compositional programs whose components can execute concurrently on varied architectures.
- Programming model: Parsl annotates functions as Apps, chains them through input and output objects, and constructs a dynamic dependency graph for concurrent execution.The model retains procedural Python code while declaring execution logic through data dependencies.
- Execution model: Parsl separates program logic from configuration and provides modular executors for laptops, clusters, supercomputers, and multiple execution models.Its architecture supports use cases ranging from low-latency to high-throughput and extreme-scale execution.
- Evaluation: 250 000 concurrent workers over 8000 nodes, greater than 1000 tasks/second, and 5ms component execution demonstrate Parsl’s scalability and performance.The reported evaluations cover a campus cluster and a supercomputer.
2 MOTIVATION
Parsl was motivated by scientific workflows that assemble heterogeneous components and require parallel execution across varied scales and interaction modes. Building on Python preserves familiar language features while supporting composable parallel programs.
- Use cases: Parsl targets workloads ranging from few long tasks to millions of short tasks by providing a generic and extensible execution model.The design addresses heterogeneous requirements rather than optimizing for one use case.
- Use cases: Scientific applications span many-task workflows, interactive notebooks, machine learning inference, online analysis, and simulations with unpredictable performance.These use cases impose differing requirements for concurrency, progress, data, and resource utilization.
- Why build on Python?: Python provides familiar control-flow constructs, libraries, and syntax that support both ordinary program logic and parallel workflow composition.Parsl preserves Python semantics where possible and can be learned without new call-site syntax.
- Why build on Python?: Parsl’s Python foundation supports composable domain libraries and application-specific scheduling behavior within the same programming language.The paper presents this as an advantage for programming both in the small and in the large.
3 DESIGN
Parsl introduces asynchronous Apps and futures that let Python programs express task dependencies while leaving execution mechanisms configurable. Its abstractions target portability, heterogeneous workloads, scalability, and reliable execution.
- Apps and futures: The App decorator and future object introduce asynchronous, parallel, and potentially remote execution into Python.Python and Bash decorators support pure Python functions and shell commands.
- Apps and futures: Invoking an App registers an asynchronous task and immediately returns a future, which later exposes results or exceptions.Futures provide blocking result access and non-blocking completion checks.
- Data handling: Parsl requires App inputs and outputs to be passed explicitly, while serialized objects and declared files support movement across execution resources.Global variables cannot be used, and immutable inputs help preserve deterministic behavior.
- Compositionality: Passing futures between Apps implicitly encodes dependencies, allowing arbitrarily large task graphs to be constructed asynchronously and scheduled according to available parallelism.This composition uses standard Python call syntax without Parsl-specific syntax at the call site.
- Execution configuration: A modular executor interface separates task transport, execution, and result communication from application logic and supports multiple execution models and providers.Configurations specify providers, executors, channels, allocations, queues, durations, and data-management options.
- Execution configuration: Parsl supports workloads from few long tasks to millions of short tasks through high-throughput, low-latency, and extreme-scale executors.The generic model is designed for varied task sizes, resource scales, and execution mechanisms.
4 ARCHITECTURE AND IMPLEMENTATION
Parsl’s DataFlowKernel maintains and orchestrates a dynamic task graph through connected executors. Its event-driven DAG execution has linear complexity in tasks and edges and includes scheduling and retry mechanisms.
- DataFlowKernel: Dependencies between Apps are represented by futures, with asynchronous callbacks allowing the DataFlowKernel to consider tasks when dependencies resolve.This makes the runtime event driven.
- DataFlowKernel: The DataFlowKernel stores task state, constructs the task graph, and manages execution through one or more connected executors.Figure 1 presents this high-level architecture.
- Task graph: O(n + e) execution time complexity results from a small fixed cost for launching each of n tasks and triggering each of e outgoing edges.The task graph is represented as a directed acyclic graph whose edges exchange input/output data.
- Scheduling: Once dependencies resolve, the DataFlowKernel schedules tasks on configured executors and associates executor futures with App futures.When multiple executors are available without execution hints, one is selected at random.
- Fault handling: Failed Apps can be retried by resubmission, while exhausted retries produce exceptions associated with the corresponding future.Failures include App execution errors and remote system failures.
4.2 Providers
Parsl uses a lightweight provider abstraction to present a uniform interface across clouds, clusters, supercomputers, grids, and workstations. Blocks, channels, and launchers translate differing resource and access models into configurable execution components.
- Provider abstraction: Parsl’s provider abstraction offers a uniform interface across clouds, supercomputers, clusters, grids, and local workstations.The abstraction is implemented in Python and supports adding providers.
- Provider abstraction: Parsl supports providers for Slurm, Torque/PBS, HTCondor, Cobalt, GridEngine, AWS, Google Cloud, Jetstream, and Kubernetes.
- Channels: LocalChannel connects directly to a local resource, whereas SSHChannel supports remote execution.Channels describe how Parsl authenticates and connects to a provider.
- Launchers: The Parsl Launcher abstracts system-specific mechanisms such as srun, aprun, and mpirun for starting workers across cores and nodes.Users may specify a launcher in the provider configuration.
- Resource representation: A block is Parsl’s basic resource unit, representing one or more nodes or a provider-specific allocation request.Blocks correspond to scheduler allocations in clusters and API requests for instances in clouds, and support elasticity.
4.3 Executors
Parsl provides modular executors for diverse workloads, sharing a common kernel while tailoring execution models to throughput, extreme scale, or latency. These executors differ in communication, distribution, and fault-tolerance mechanisms.
- Executor design: Parsl provides a modular executor interface because Apps range from milliseconds to days and available parallelism ranges from none to millions.Executors target workstations, clouds, and supercomputers.
- Executor design: Parsl executors extend Python’s concurrent.futures interface with automatic scaling, monitoring, deferred initialization, and working-directory support.
- Common kernel: All executors share a kernel that deserializes Apps and inputs, then runs tasks in a sandboxed Python environment.
- High Throughput Executor: HTEX uses a pilot-job model with an executor client, interchange, and managers to support high-throughput task execution.Managers batch and prefetch tasks, while the interchange brokers work according to advertised capacity.
- Extreme Scale Executor: EXEX uses MPI and hierarchical task distribution for multi-node execution on extreme-scale supercomputers.Rank 0 acts as manager and the remaining MPI ranks act as workers.
- Low Latency Executor: LLEX reduces latency by minimizing execution features and removing one communication hop between workers and the interchange.It sacrifices reliability and automated provisioning, and assumes fixed resources because provisioning can take seconds to minutes.
4.4 Elasticity
Parsl addresses changing workload parallelism with elastic resource blocks that grow or shrink in response to waiting work. Users can configure or replace the strategy governing this behavior.
- Motivation: Reserving resources for peak parallelism can underutilize them, while reserving minimal resources can extend execution time.This trade-off also occurs in simple bag-of-task applications with tasks of different durations.
- Elasticity model: Parsl provisions and deprovisions resource blocks in response to workload pressure through an extensible elasticity strategy interface.The default strategy uses a configurable parallelism parameter to control how aggressively resources grow and shrink.
4.5 Data management
Parsl’s data manager preserves execution-location independence by representing files abstractly and transferring them to the resources where Apps need them. Transfers can become dynamic dependencies in the task graph.
- File abstraction: Parsl’s file abstraction avoids hard-coded paths by transparently transferring files and translating their physical locations.The data manager handles file movement between Apps and execution resources.
- Protocols: Parsl supports local files and remote files accessed through HTTP, FTP, or Globus.
- Dynamic dependencies: When a needed remote file is unavailable, Parsl creates a data-transfer task and a dynamic dependency before executing dependent Apps.
- Transfer execution: HTTP and FTP transfers execute through the executor, while Globus transfers can run directly through the data manager.Globus staging can defer resource provisioning until data has been staged.
4.6 Additional features
Parsl provides supporting features for secure, portable, observable, reproducible, and customized parallel execution.
- Authentication: Authentication integrates Globus Auth for interactive or token-based access to enabled services.Tokens can support data transfers and SSH access to compute resources.
- Containers: Containers let workers or individual task invocations run inside predefined environments.This packages software for customized execution across heterogeneous environments.
- Monitoring: Monitoring logs task states, execution metadata, resource usage, and supports storage in databases, Elastic Search, or files.A web interface provides access to recorded execution information.
- Memoization: Memoization avoids repeated execution of Apps with identical inputs at program or individual-App scope.It is configurable because memoization is rarely useful for non-deterministic Apps.
5 EVALUATION
The evaluation measures Parsl’s latency, scaling, worker capacity, throughput, and elastic resource utilization on Midway and Blue Waters. Results show low latency, large-scale execution, strong scaling performance, and higher utilization with elasticity, with stated platform and allocation boundaries.
- 5.1 Latency: 3.47 ms average latency makes LLEX faster and less variable than the other tested executors.Parsl executors were faster than IPP at 11.72 ms and Dask at 16.19 ms average latency.
- 5.2 Scalability: HTEX provides the best strong-scaling performance, while EXEX scales to significantly more workers than the other executors and frameworks.Both remain nearly constant across the tested worker counts; FireWorks has almost an order of magnitude greater overhead despite using 5000 tasks.
- 5.2 Scalability: HTEX and EXEX outperform the other executors and frameworks in weak scaling beyond 4096 workers.FireWorks becomes sublinear around 32 workers, IPP at 256, and Dask, HTEX, and EXEX at 1024 workers.
- 5.2 Scalability: 262 144 workers were connected with EXEX across 8192 Blue Waters nodes, while HTEX reached 65 536 workers across 2048 nodes.These measurements were limited by the available Blue Waters allocation for HTEX and EXEX.
- 5.3 Throughput: 2617 tasks/s was the highest measured throughput, achieved by Dask distributed; HTEX and EXEX achieved 1181 and 1176 tasks/s.The throughput experiment ran 50 000 no-op tasks on Midway with varying worker counts.
- 5.4 Elasticity: 84.28% average worker utilization with elasticity exceeded 68.15% without elasticity, while makespan increased from 301 s to 331 s.The elasticity study used a four-stage workflow with wide parallel stages and single-task reduce-like stages.
6 RELATED WORK
Parsl differs from specialized analytics, machine-learning, and workflow systems by augmenting Python itself with dynamic, general-purpose parallelism. Its task-graph model spans short and long tasks across execution scales while retaining Python’s programming flexibility.
- Analytics and workflow systems: Parsl augments Python with parallelism, whereas Hadoop and Spark primarily target I/O-intensive map-reduce analytics.Parsl is presented as a more general approach that enables various types of parallelism in Python.
- Analytics and workflow systems: Static-DAG systems require workflows to be defined upfront, while Parsl uses Python to create dynamic parallel programs.Pegasus, Galaxy, and CWL use static workflow descriptions; Parsl offers the full Python language for dynamic parallelism.
- Analytics and workflow systems: Custom-DSL systems such as Swift and NextFlow offer performance but impose steep learning curves and limited programming constructs.Their specialized languages differ from Parsl’s strategy of extending familiar Python code.
- Python workflow systems: Python workflow systems differ in emphasis: FireWorks prioritizes fault tolerance, Airflow uses centralized scheduling, and Luigi represents parallelism through classes.These systems explicitly describe or introspect dependency graphs rather than augmenting Python with parallelism in Parsl’s manner.
- Parallel Python systems: Dask focuses on parallel Python libraries and small-cluster distributed execution, while Ray targets reinforcement-learning training, serving, and simulation.The cited systems address narrower application or deployment settings than Parsl’s broader parallelism model.
- Parallel Python systems: Parsl targets a broader range of Python parallelism, from many short tasks through long tasks at extreme scale.The paper positions its underlying task-graph model as applicable to comparable frameworks seeking higher-level parallelism.
7 SUMMARY
Parsl expresses safe, deterministic parallelism through simple Python decorators and supports scalable execution across heterogeneous resources. Its flexible executor model addresses varied workloads, while ongoing work expands data management and higher-level synchronization.
- 7 SUMMARY: Parsl uses simple decorators to express safe, deterministic parallel programs while supporting execution from laptops to supercomputers.Its architecture is intended to address varied requirements of scientific analyses.
- 7 SUMMARY: ∼8000 nodes, less than 5 ms execution overhead, and ∼1200 tasks per second characterize Parsl’s reported scalable execution envelope.The paper presents these figures as outcomes of its performance studies and provides executor-selection guidelines.
- 7 SUMMARY: Parsl’s executor model fills varied use cases spanning concurrent long-running tasks, short-task execution, and large-scale deployments.The comparison contrasts Parsl with systems focused on fewer than 1000 long tasks or fewer than 100 nodes.
- 7 SUMMARY: Future work targets map constructs, synchronization primitives, higher-level domain libraries, direct data staging, ephemeral caching, and optional sandboxing.These plans expand Parsl’s parallelism and data-management capabilities.