Source-linked AI summary
AiiDA: Automated Interactive Infrastructure and Database for Computational Science
Giovanni Pizzi, Andrea Cepellotti, Riccardo Sabatini, Nicola Marzari, Boris Kozinsky
TL;DR
Computational science needs infrastructure to organize increasingly extensive simulations while improving reproducibility, searchability, and sharing. The paper presents the four-pillar ADES model and its implementation in the open-source AiiDA platform, whose graph-based provenance model, automated resource management, and workflow support integrate calculations with their data. AiiDA demonstrates this approach through reproducible workflow execution, while data-format standardization remains outside its scope.
Problem
Computational science is increasingly extensive, but its organization, preservation, reproducibility, and sharing require infrastructure beyond individual artisanal practices.
Method
The paper defines four infrastructure pillars—Automation, Data, Environment, and Sharing—and implements them in AiiDA using graph-based provenance, coupled storage and automation, plugins, and scientific workflows.
Results
AiiDA reproduced the published CMST results for the PSlibrary pseudopotential family using a Quantum ESPRESSO workflow with iterative fitting and automatic restart.
Takeaways & Limitations
AiiDA provides an integrated open-source infrastructure for preserving, querying, reproducing, automating, and sharing computational simulations and their workflows.
Takeaways & Limitations
Standardizing formats produced by different codes is outside AiiDA’s scope, although default plugins support common structures and file-format exchange.
Abstract
from arXiv · showhide
Computational science has seen in the last decades a spectacular rise in the scope, breadth, and depth of its efforts. Notwithstanding this prevalence and impact, it is often still performed using the renaissance model of individual artisans gathered in a workshop, under the guidance of an established practitioner. Great benefits could follow instead from adopting concepts and tools coming from computer science to manage, preserve, and share these computational efforts. We illustrate here our paradigm sustaining such vision, based around the four pillars of Automation, Data, Environment, and Sharing. We then discuss its implementation in the open-source AiiDA platform (http://www.aiida.net), that has been tuned first to the demands of computational materials science. AiiDA's design is based on directed acyclic graphs to track the provenance of data and calculations, and ensure preservation and searchability. Remote computational resources are managed transparently, and automation is coupled with data storage to ensure reproducibility. Last, complex sequences of calculations can be encoded into scientific workflows. We believe that AiiDA's design and its sharing capabilities will encourage the creation of social ecosystems to disseminate codes, data, and scientific workflows.
1. Introduction
Computational science has expanded into high-throughput simulations, creating a need for infrastructure that organizes simulations and data while balancing flexibility, usability, reproducibility, and sharing. AiiDA addresses these needs through an integrated platform built around four pillars: Automation, Data, Environment, and Sharing.
- Motivation: Growing simulation capacity and curated materials databases have enabled high-throughput materials design and discovery.Quantum-mechanical simulation codes are now routinely used in science, technology, industry, and academia.
- Challenge: Automating and managing simulations and their data requires infrastructure that is both flexible across tasks and easy to use.The same infrastructure must also address reproducibility and community sharing.
- AiiDA: AiiDA provides transparent access to local and remote resources, supports different codes through plugins, and couples storage with workflow automation.Its high-level Python interface is designed to reduce user effort while supporting heterogeneous computational tasks.
- AiiDA: Directed acyclic graphs represent data, calculations, and provenance relationships for reproducible storage, querying, and analysis.The graph-based database design targets high-throughput simulations and heterogeneous results.
- ADES model: The ADES model organizes the infrastructure around Automation, Data, Environment, and Sharing.The user-facing environment and social ecosystem support sharing codes, data, and workflows.
2. The ADES model for computational science
The ADES model defines requirements for infrastructure that automates simulations, manages heterogeneous data and provenance, provides a usable environment, and supports sharing. Its pillars connect low-level computational services with reusable workflows, analytics, remote management, and community standards.
- Automation: Automation abstracts preparation, submission, retrieval, and storage of large numbers of calculations.High-throughput automation can run calculations simultaneously, analyze and filter results, and handle errors when possible.
- Automation: Remote management uses a common API and plugins to automate workstation-to-HPC job preparation, submission, status checks, and retrieval.Communication protocols and schedulers can vary behind the shared interface.
- Automation: Tightly coupling automation and storage supports reproducibility by pre-storing simulation information and automatically creating consistent calculation pipelines.This reduces manual uploading of inputs and outputs and limits incomplete repository information.
- Data: The Data pillar requires permanent storage for relevant heterogeneous outputs, flexible attributes, and efficient queries across data and provenance relationships.AiiDA uses a directed acyclic graph model and a relational backend with transitive-closure support for graph traversal.
- Environment: Environment features include API abstractions, high-level programming, reusable scientific workflows, and interfaces to analytics tools.Workflows encode dependencies between calculation steps, including dependencies determined by intermediate results.
- Sharing: Sharing depends on social ecosystems, access policies, plugins, standard formats, and community-driven ontologies for exchanging data and workflows.Repository pipelines can import or export data and provenance through defined interfaces and formats.
3. The AiiDA infrastructure
AiiDA is an open-source Python infrastructure positioned between researchers and HPC resources, providing an abstraction layer for automating, storing, managing, and sharing simulations and their results. Its API and plugin architecture hide differences among storage systems, computers, codes, data types, and schedulers.
- Implementation: AiiDA implements the ADES requirements as a distributed open-source Python infrastructure.The implementation focuses on managing computational research efforts and meeting the model’s infrastructure requirements.
- Architecture: AiiDA acts as an intermediate layer between users and HPC resources, achieving automation through abstraction.This separates user interaction from details of remote computational resources.
- API: The API exposes calculations, codes, and data through Python classes while hiding differences among supercomputers and storage solutions.Its ORM maps stored objects into Python classes and provides high-level creation, modification, and query operations.
- Extensibility: AiiDA’s plugin architecture extends support for simulation codes, data types, and remote computers with different schedulers.Users can interact through the command-line tool, an interactive Python shell, or Python scripts.
4. Automation in AiiDA
AiiDA automates calculation lifecycles through a background daemon and abstract transport and scheduler interfaces. The daemon submits jobs, monitors scheduler states, retrieves and parses results, and updates calculation states without requiring direct remote access.
- Daemon: The daemon runs in the background to coordinate HPC-cluster interactions and AiiDA’s automation tasks.It uses communication-channel and scheduler plugins and can continue operating after users log out.
- Daemon: The calculation lifecycle consists of remote submission, scheduler-state verification, and result retrieval and parsing.These management operations are implemented independently and can group calculations running on the same machine.
- Monitoring: Calculation states are stored in the database and continuously updated by the daemon, allowing users to monitor jobs without connecting directly to remote machines.Submitting a calculation advances it through states such as NEW, TOSUBMIT, and WITHSCHEDULER.
- Architecture: Figure 2 links the API, ORM, plugin interface, daemon, communication channels, and scheduler plugins in AiiDA’s automation architecture.The daemon handles job submission, scheduler checks, file retrieval, and parsing across local and remote clusters.
- Transports and schedulers: An abstract transport and scheduler API keeps AiiDA’s internal code and user interface independent of specific connection protocols and schedulers.Generic job attributes use a common format, while plugins convert abstract information into scheduler-specific submissions.
5. Data in AiiDA: database, storage and provenance
AiiDA represents calculations, codes, and data as a directed acyclic graph whose labeled links preserve provenance, while a flexible SQL-backed schema stores heterogeneous attributes and supports efficient querying.
- Data model and provenance: Calculations act on input data through executable Code nodes and produce output Data nodes that can feed subsequent calculations.Direct Data-to-Data links are disallowed; transformations are represented by intermediate Calculation nodes.
- Data model and provenance: Directed acyclic graphs preserve the causal chain from initial data through intermediate steps to final results.Acyclicity reflects the requirement that links represent causal connections rather than loops.
- Database and storage: AiiDA stores DAG nodes, links, and attributes in SQL tables complemented by a file repository for arbitrary files and directories.The main tables are DbNode, DbLink, and DbAttribute; the repository handles data that does not require direct querying.
- Database and storage: The DbAttribute EAV design stores heterogeneous primitive, list, and dictionary values while preserving direct database-level queryability.Nested list and dictionary elements can also be queried efficiently.
- Database and storage: Immutable stored-node attributes are supplemented by DbExtra for user-defined attributes used in later search and filtering.Attributes become unmodifiable after permanent storage, while custom metadata can be kept separately.
- Provenance queries: The DbPath table accelerates history queries, with storage growth remaining roughly linear for many small materials-science graphs.This design was selected as a compromise between query speed and disk usage.
6. The scientific environment in AiiDA
AiiDA provides a Python-centered scientific environment with ORM, command-line, shell, and scripting interfaces, extensible plugins, and a workflow engine for repeated multistep computations.
- User environment: The Python ORM exposes calculations, codes, and data through an intuitive interface while hiding database and storage details.The Node class provides access to attributes, repository files, links, and queries.
- Extensibility: Plugins extend support for new calculations, data types, simulation codes, transport channels, and schedulers.A plugin is a Python module defining subclasses that AiiDA detects automatically.
- User environment: Users can interact with AiiDA through the verdi command-line tool, Python scripts, or an interactive Python shell.The shell and verdi run can automatically load the AiiDA database environment.
- Scientific workflows: The workflow engine encodes repeated multistep tasks and can generate dependent calculations based on intermediate results.Users can define generic workflows and application-specific automation schemes through the API.
7. Sharing in AiiDA
AiiDA supports sharing through private local instances, selective database export and import, common importer/exporter APIs, and reusable data formats and workflows.
- Controlled sharing: Researchers can keep data private locally while exporting selected database portions for collaborators or project repositories.Separate local instances and group access policies support controlled sharing and authorship preservation.
- Controlled sharing: AiiDA assigns each node a UUID to prevent identity conflicts when data moves between database instances.Auto-incrementing primary keys are not preserved across transfers, so UUIDs provide stable node identity.
- Interoperability: Common data subclasses and importer/exporter plugins allow exchange with different codes, file formats, infrastructures, and repositories.AiiDA provides reusable structures such as crystal structures and k-point paths, with support for external databases.
- Interoperability: Standardizing formats produced by different codes remains outside AiiDA’s scope, although the platform encourages such efforts and supports high-level workflows meanwhile.Workflows can perform tasks such as structure optimization or molecular dynamics before format standardization is complete.
8. Codes and data types supported out of the box
AiiDA initially emphasizes infrastructure while providing functional Quantum ESPRESSO plugins, several standardized data subclasses, and support for additional code ecosystems.
- Initial scope: The initial development focused mainly on infrastructure components such as scheduler management and the database rather than many plugins.This scope choice preceded the broader collection of supported codes and data types.
- Code support: Fully functional plugins support major Quantum ESPRESSO codes and post-processing tools, including input generation and output parsing.Supported examples include pw.x, ph.x, cp.x, matdyn.x, and q2r.x.
- Data support: AiiDA supplies standardized Data subclasses for crystal structures, reciprocal-space k-point paths, band structures, and generic arrays.Examples include StructureData, KpointsData, BandsData, and ArrayData.
- Code support: Additional support covers codtools, GPAW, and ASE-connected codes such as VASP, ABINIT, and Gaussian.ASE enables the same plugin to access a larger family of simulation codes.
9. Examples of applications
AiiDA is applied to automate verification, benchmarking, high-throughput screening, and repository sharing in computational materials science. These applications use reproducible workflows and provenance to reduce intervention and support reuse of results.
- 9.1. Verification and validation: AiiDA workflows automate verification tests, identify calculations using inaccurate pseudopotentials, and enable their reproducible reruns with improved inputs.The pseudopotential workflow performs standardized tests with minimal intervention and uses stored provenance to find affected calculations.
- 9.1. Verification and validation: The CMST workflow iteratively refines a lattice parameter by fitting a Birch–Murnaghan equation of state to 15 volume-dependent calculations.It uses Quantum ESPRESSO and includes an automatic restart subworkflow for crashed calculations.
- 9.2. Code benchmarking: AiiDA supports benchmarking multiple code or method versions and parametric sweeps for accuracy checks, performance recording, uncertainty quantification, and transferability assessment.These workflows record information needed to reproduce results while collecting statistics on training structures.
- 9.3. High-throughput material screening: Automated workflows explore materials spaces such as ABO3 perovskites by evaluating finite-temperature properties through equations of state and phonon spectra at different volumes.The workflow performs these intermediate calculations for every material.
- 9.4. Open repositories: AiiDA captures provenance, simplifying reproducible uploads to cloud repositories and existing open databases such as TCOD.Automated exporters can provide computed structures together with the full calculation tree that generated them.
10. Conclusions
The paper presents AiiDA as an implementation of four infrastructure pillars—Automation, Data, Environment, and Sharing—using a high-level API and graph-oriented storage for computational science.
- 10. Conclusions: AiiDA implements the four infrastructure pillars of Automation, Data, Environment, and Sharing through a high-level Python API and ORM.The API exposes Calculation, Data, and Code objects while remaining independent of the detailed storage solution and SQL query language.
- 10. Conclusions: Its storage combines repository folders with an SQL database supporting directed acyclic graphs, general attributes, heterogeneous data, and optimized graph traversal.The supported SQL backends include PostgreSQL, MySQL, and SQLite, while entity-attribute-value tables accommodate heterogeneous attributes.
Appendix A.1. Attributes in a EAV table
The EAV design stores variable attributes in SQL while supporting nested lists and dictionaries through path-like keys and datatype metadata. A separate disk-backed class is recommended for large arrays.
- Appendix A.1. Attributes in a EAV table: Relational schemas can limit applications requiring variable properties because SQL tables define a fixed number and type of columns.This motivates a more flexible attribute representation.
- Appendix A.1. Attributes in a EAV table: AiiDA uses modified entity-attribute-value tables to store heterogeneous attributes while retaining SQL-based storage and querying.The backend choice balances usability, storage speed and size, and query efficiency.
- Appendix A.1. Attributes in a EAV table: Serializing lists and dictionaries as JSON strings makes searching specific list values inefficient, so AiiDA stores nested structures through separate typed entries.The alternative schema supports lists and dictionaries at arbitrary depth and preserves datatypes such as dates and None.
- Appendix A.1. Attributes in a EAV table: Dictionary entries record the container type and element count, while nested item keys are formed by joining the dictionary path with a dot separator.Lists use integer positions instead of dictionary item names.
- Appendix A.1. Attributes in a EAV table: The nested EAV storage model supports arbitrary depth and efficient querying of small lists, but large arrays should be stored on disk with ArrayData.The paper explicitly limits this model to small lists and dictionaries.
Appendix A.2. Algorithm for the update of the transitive closure table
The transitive-closure algorithm maintains graph connectivity by recording paths between linked nodes and updating affected ancestor–descendant combinations whenever a link changes.
- Appendix A.2. Algorithm for the update of the transitive closure table: Each transitive-closure entry represents connectivity between two graph nodes through a sequence of links.The table therefore records path relationships rather than only direct links.
- Appendix A.2. Algorithm for the update of the transitive closure table: Database triggers automatically invoke updates whenever an entry is added, removed, or modified in the link table.This keeps the closure table synchronized with graph-link changes.
- Appendix A.2. Algorithm for the update of the transitive closure table: When a link A→B is created, the algorithm adds paths from A and its parents to B and its children, including every parent–child combination.Existing closure paths are queried first to obtain A’s parents and B’s children.