Source-linked AI summary
AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance
Sebastiaan. P. Huber, Spyros Zoupanos, Martin Uhrin, Leopold Talirz, Leonid Kahle, Rico Häuselmann, Dominik Gresch, Tiziano Müller, Aliaksandr V. Yakutovich, Casper W. Andersen, Francisco F. Ramirez, Carl S. Adorf, Fernando Gargiulo, Snehal Kumbhar, Elsa Passaro, Conrad Johnston, Andrius Merkys, Andrea Cepellotti, Nicolas Mounet, Nicola Marzari, Boris Kozinsky, Giovanni Pizzi
TL;DR
Computational science needs scalable automation and provenance tracking as calculations and interconnected data grow toward exascale workloads. AiiDA 1.0 combines an event-based workflow engine, queryable provenance infrastructure, Python workflows, and plugins, sustaining tens of thousands of processes per hour while preserving traceability. Its scope is bounded by the limitations of atomic, blocking work functions for long-running workflows, which WorkChains address through resumable steps.
Problem
Growing computational workloads and interconnected data make a posteriori provenance reconstruction intractable, creating a need for automated, scalable, and reusable workflow infrastructure.
Method
AiiDA 1.0 combines an automated engine, Python workflow language, provenance graph and query builder, relational storage, and plugin system for simulation-code interoperability.
Results
AiiDA 1.0 sustains tens of thousands of processes per hour across local computers and large high-performance supercomputers while automatically preserving full provenance.
Takeaways & Limitations
AiiDA’s provenance graph, workflow automation, and plugin registry support reproducible, shareable, and interoperable computational workflows across research fields.
Takeaways & Limitations
Work functions are atomic and blocking, so interruptions during complex workflows can lose intermediate progress; WorkChains provide resumable step-based execution.
Abstract
from arXiv · showhide
The ever-growing availability of computing power and the sustained development of advanced computational methods have contributed much to recent scientific progress. These developments present new challenges driven by the sheer amount of calculations and data to manage. Next-generation exascale supercomputers will harden these challenges, such that automated and scalable solutions become crucial. In recent years, we have been developing AiiDA (http://www.aiida.net), a robust open-source high-throughput infrastructure addressing the challenges arising from the needs of automated workflow management and data provenance recording. Here, we introduce developments and capabilities required to reach sustained performance, with AiiDA supporting throughputs of tens of thousands processes/hour, while automatically preserving and storing the full data provenance in a relational database making it queryable and traversable, thus enabling high-performance data analytics. AiiDA's workflow language provides advanced automation, error handling features and a flexible plugin model to allow interfacing with any simulation software. The associated plugin registry enables seamless sharing of extensions, empowering a vibrant user community dedicated to making simulations more robust, user-friendly and reproducible.
INTRODUCTION
AiiDA 1.0 addresses computational reproducibility and high-throughput workflow management by automatically recording provenance, supporting scalable execution, and providing flexible workflow abstractions. Its provenance model and Python-based workflows enable traceability, automation, interruption recovery, and reusable higher-level workflows.
- Motivation and architecture: AiiDA 1.0 redesigns the engine to manage high-throughput computational workloads across local and remote resources.The engine automates calculations and workflows and can distribute concurrent tasks across multiple computational resources.
- Motivation and architecture: AiiDA records calculations, workflows, inputs, and outputs in provenance graphs so computational results can be exactly retraced.The engine automatically represents process execution and its inputs and outputs as graph nodes and links.
- Workflow automation: Work chains address the atomic, blocking execution of work functions by persisting progress between workflow steps.If a work function is interrupted, intermediate progress cannot be persisted and all progress is lost; work chains are designed to stop and continue execution.
- Workflow automation: The Python workflow language supports complex sequences with nested subprocesses, error handling, restarting, and parameter tuning.Work chains divide execution into engine-managed steps, allowing control to return between subprocesses and enabling interruption and continuation.
- Workflow automation: Python-native workflows avoid translation between user code and engine execution, facilitating debugging and integration with external data-manipulation libraries.The design lets users use the full Python and AiiDA APIs when defining workflows.
- Provenance model: AiiDA’s node hierarchy distinguishes data and process nodes, while separate data- and workflow-focused provenance layers support different inspection granularities.The data-and-calculation subgraph is a DAG, whereas the logical provenance graph includes workflows and may contain cycles.
“Database abstraction and querying language”).
AiiDA combines immutable node storage, provenance tracking, relational-graph querying, and a Python query builder that translates requests into optimized SQL. These tools support reproducibility, caching, and analysis of very large provenance graphs.
- Data model: Immutable attributes and repository files jointly define each node’s content, preventing changes that would invalidate provenance.Queryable properties are stored as attributes, while large or non-queryable data are stored in the repository.
- Provenance: Recording process transformations and inputs produces a provenance graph that can reproduce results when all relevant inputs and code are stored.AiiDA stores process-function source code and references other process code through AiiDA and plugin versions.
- Caching: Identical process inputs can reuse previously computed outputs, allowing AiiDA’s caching mechanism to skip repeated execution.The cache avoids incurring the computational cost of executing the same process again.
- Database abstraction: AiiDA stores provenance graphs and most node properties in a relational database so tools can query graphs containing millions of nodes.Efficient graph queries are needed for data analysis in large automated high-throughput projects.
- Querying language: The query builder expresses provenance-graph queries in Python, translates them into optimized SQL, and searches for matching subgraphs.Queries can constrain link directions, link types, node types, and node properties such as chemical elements or value ranges.
THE REST API
AiiDA provides a secure REST API for querying and serving provenance data over the web. Its endpoints support scalable retrieval, node-specific access, and interactive browsing through Materials Cloud Explore.
- Motivation: Web access requires a secure solution that can query selected data rather than exposing only the complete database.This complements direct Python-interface querying when users lack full access to the AiiDA machine.
- API implementation: AiiDA’s REST API server can run from the command line or behind scalable web servers and exposes graph data, node properties, and repository files.The API maps query results to REST formats and serializes them into JSON responses.
- API implementation: Paginated results support large downloads without overloading the server, while subtype-specific endpoints provide raw calculation inputs, outputs, and formatted data.Common node endpoints are supplemented by functionality tailored to particular node subtypes.
- Use case: Materials Cloud Explore uses the REST API to present AiiDA databases as interactive pages with graph browsing through a graphical interface.The API provides the data needed to display provenance graphs and node contents.
Registry
AiiDA’s plugin registry improves discovery and reuse of extensions, while community practices support reproducibility and long-term data access. Data longevity is addressed through migrations and transitional Python compatibility.
- Registry: The online plugin registry makes packages discoverable and encourages code sharing and reuse across the AiiDA community.Authors register package metadata, documentation, code links, and a developer-maintained JSON description.
- Community building: Community reproducibility requires coordinated effort beyond releasing software, including robust code, durable data, and user and developer training.AiiDA supports uptake through tutorials, workshops, and participation in NumFOCUS.
- Data longevity: Data longevity means accessing data created by earlier code versions from newer versions, despite performance-driven changes to data layouts.This boundary is especially important because AiiDA targets reproducible, high-throughput simulations.
- Data longevity: 42 migrations per backend in AiiDA 1.0 enable automatic database upgrades across versions without losing data or provenance.Archive files require separate Python migrations because database migrations cannot be reused for data-only archives.
- Data longevity: AiiDA 1.0 supports Python 2 and Python 3, providing a six-month transition period after Python 2 reached end-of-life.The compatibility period was intended to help users and developers upgrade code and plugins to Python 3.
Interoperability
AiiDA extends interoperability through plugins, data-conversion interfaces, database imports, dissemination tools, and community practices that support reproducible computational research.
- AiiDA’s plugin system interfaces with various simulation codes while centering interoperability on provenance-based reproducibility.
- Built-in interfaces support ASE, pymatgen, spglib, seekpath, and imports from ICSD, COD, TCOD, and OQMD.
- Materials Cloud integration enables interactive graphical exploration of AiiDA databases.
- Online tutorials, recordings, virtual machines, documentation, and developer support broaden access to AiiDA and its plugins.
- By March 2020, the plugin registry hosted 47 packages covering over 90 simulation codes and around 80 workflows.
- AiiDA Enhancement Proposals provide a standardized, reviewable process for community suggestions and future extensions.
METHODS
AiiDA’s methods use independent daemon runners, persistent task queues, and event-based communication to execute, supervise, and control concurrent workflows.
- Each launched process is assigned to a runner that executes it and persists its state as a checkpoint for restartability.The checkpoint serializes the process instance to YAML and stores it in the database.
- Independent daemon runners execute in separate system processes and communicate through a message broker, enabling parallel operation without database concurrency issues.
- A single daemonized controller starts, stops, monitors, and scales the number of active runners.
- RabbitMQ task queues persist submitted processes and redistribute tasks when a runner becomes unreachable.
- Separate communication threads protect heartbeat handling and schedule database callbacks on the runner’s event loop.
- Dedicated process channels support remote pause, restart, and kill operations, while broadcasting state changes to subscribed listeners.
EAV replaced by JSONB
AiiDA replaces storage-intensive EAV attributes and explicit transitive-closure tables with JSONB storage and on-the-fly graph traversal, improving query and storage efficiency under typical workloads.
- EAV replaced by JSONB: The EAV schema supports arbitrary nested attributes but increases storage, serialization overhead, and query cost.
- EAV replaced by JSONB: JSONB reduces transfer and deserialization costs through a more compact schema than EAV.
- EAV replaced by JSONB: 75 times smaller total query time was measured for site attributes with JSONB than with EAV.The benchmark used PostgreSQL 10.10 and 300,000 crystal-structure nodes from a database containing 7,318,371 nodes.
- On-the-fly transitive closure: An explicit transitive-closure table can reach 226 million rows and 200GB for one million provenance nodes.
- EAV replaced by JSONB: The JSONB site query’s SQL time was roughly 6.5 times smaller than the equivalent EAV query.
- On-the-fly transitive closure: On-the-fly transitive closure removes closure-table storage and accelerates new-link insertion, while making recursive queries slightly slower.
- On-the-fly transitive closure: For graph topologies typical of AiiDA provenance, the on-the-fly approach has minimal recursive-query efficiency impact and is about twice slower than the table-based approach.The benchmark caveat applies to the tested binary-tree topologies comparable to typical AiiDA provenance graphs.
Event versus polling-based engine
AiiDA’s event-based engine improves process completion over the earlier polling-based engine, despite slightly slower submission. Its optimized configuration also scales through dynamically increased daemon-worker concurrency.
- The event-based engine executes processes continuously, whereas the polling-based engine finalizes them in batches.
- The new engine completes processes faster than the old engine, even under constrained conditions.
- Three times faster completion is achieved by the optimized new engine than by the old engine in the arithmetic work-chain example.
- The daemon can dynamically increase its worker count to distribute heavy workloads.
- The optimized new engine, using 12 workers without connection delay, completes all processes in half the time required by its constrained configuration.
Caching
AiiDA caching avoids repeating calculations with identical inputs and allows corrected workflows to resume effectively from prior successful computations.
- Caching reuses outputs when a calculation with exactly the same inputs has already been completed, saving computational resources.
- AiiDA hashes each immutable node’s complete content and searches for an existing node with the same hash before launching a calculation.
- When caching is enabled, a failed workflow can be rerun after correction without repeating successful calculations.
CODE AVAILABILITY
AiiDA’s source code and database-performance materials are openly distributed through public software and research-data repositories.
- AiiDA’s source code is available under the MIT open-source license on GitHub and as an installable Python Package Index package.
- The data used for Figs. 6–7 and Table I are available through the Materials Cloud Archive.
- The data and analysis scripts used for Fig. 8 are available through the Materials Cloud Archive.
Supplementary Information: AiiDA 1.0, a scalable computational infrastructure for
The supplementary information concerns AiiDA 1.0 as a scalable infrastructure for automated reproducible workflows and data provenance.
- AiiDA 1.0 is presented as a scalable computational infrastructure for automated reproducible workflows and data provenance.
A. Architecture differences with earlier AiiDA versions
AiiDA 1.0 redesigns the infrastructure for scalable high-throughput HPC workflows while preserving provenance and improving extensibility, interfaces, and data access. The new architecture combines a scalable event-based engine, integrated provenance representation, query tools, REST access, and an independent plugin system.
- Architecture and scalability: AiiDA 1.0 substantially redesigns the 0.x codebase to support scalable high-throughput HPC workloads and future exascale machines.The redesign also targets flexibility, simpler extension protocols, and improved usability.
- Architecture and scalability: The event-based engine replaces polling, reacts immediately to state changes, and supports parallel workers for sustained high process throughput.It also adds automatic rescheduling for transient failures and pauses calculations after repeated consecutive failures.
- Provenance and data access: Calculations and workflows now share a homogenized interface, while workflows are explicitly represented in the PostgreSQL provenance graph.This integration retains the provenance graph while making workflow links explicit.
- Provenance and data access: The QueryBuilder expresses provenance-graph queries in Python and translates them into SQL through SQLAlchemy for efficient data and provenance retrieval.The supplementary example uses graph relationships, filters, and projections to retrieve relaxation thresholds and energies.
- Interfaces and extensibility: A REST API enables programmatic HTTP(S) access to the provenance graph, including custom interfaces such as the Materials Cloud graphical interface.The CLI was also rebuilt around Click for consistent commands and reusable components.
- Interfaces and extensibility: The new plugin system decouples plugin development from AiiDA’s core source tree, while a registry lets users discover shared extensions.This replaces the earlier tightly coupled plugin-development model.
B. Query builder syntax example
The QueryBuilder example shows how AiiDA specifies a provenance subgraph, applies node and relationship filters, and projects selected properties as returned results. The query targets energies from calculations using a named code and relaxation data.
- Query target: The example queries calculations that compute a crystal structure’s total energy after relaxation within a specified threshold using code labeled my-code.Its goal is to obtain total energy as a function of relaxation threshold.
- Query construction: A QueryBuilder object is built by appending nodes, declaring filters and inter-node relations, and specifying projections for returned properties.This syntax defines both the matched graph structure and the values extracted from matching nodes.
- Query construction: The query matches a CalcJobNode with a my-code input and a Dict input whose type attribute is relax, projecting the threshold attribute.The calculation’s result dictionary contributes the energy projection through the results-labeled incoming link.
- Query results: qb.all() returns one ordered pair of projected values for each matching subgraph, producing results in the form [(threshold1, energy1), (threshold2, energy2), ...].Each pair contains the relaxation threshold and computed energy.
C. Example of a work chain and calculation function
The Fibonacci example demonstrates how AiiDA’s workflow language combines persistent workflow state, iterative logic, and calculation functions while automatically recording provenance. Each addition and its intermediate result becomes part of the execution graph.
- Workflow implementation: The example implements a Fibonacci calculator with a work chain and an addition calculation function.The work chain follows the Fibonacci recurrence and delegates each addition to the calculation function.
- Workflow implementation: The work chain initializes iteration and Fibonacci-state variables, then uses a persisted context to transfer values between logical steps.The context stores variables such as the iteration counter and the previous and current Fibonacci numbers.
- Workflow implementation: Each iteration sums the previous and current values, corresponding directly to Eq. (S1), until N − 1 iterations have been completed.After the iterations, the current value is returned as the requested Fibonacci number.
- Provenance recording: For N = 5, the execution provenance graph records the work chain, initial input, final result f5 = 5, individual additions, and intermediate results.The example illustrates that arbitrary workflow logic can be implemented while provenance is automatically stored.