Source-linked AI summary
A Data Ecosystem to Support Machine Learning in Materials Science
Ben Blaiszik, Logan Ward, Marcus Schwarting, Jonathon Gaff, Ryan Chard, Daniel Pike, Kyle Chard, Ian Foster
TL;DR
Researchers lack easy access to high-quality materials data, limiting the data ecosystem needed for machine learning and materials discovery. The paper presents MDF and DLHub capabilities that connect materials data services and support streamlined discovery and dissemination of machine-learning results.
Problem
Researchers lack easy access to high-quality materials data even as the volume of available data grows, while ML opportunities require more and better data.
Method
The paper presents MDF and DLHub capabilities as interconnected, modular infrastructure services for connecting materials data and supporting materials discovery.
Results
MDF and DLHub capabilities can streamline complex materials discovery and deliver machine-learning study results to a broad audience.
Takeaways & Limitations
These capabilities are presented as critical to building a materials data ecosystem optimized for widespread application of machine learning.
Takeaways & Limitations
The ecosystem still requires broader, simpler dissemination and discovery of datasets across hundreds of common materials-science file types.
Abstract
from arXiv · showhide
Facilitating the application of machine learning to materials science problems will require enhancing the data ecosystem to enable discovery and collection of data from many sources, automated dissemination of new data across the ecosystem, and the connecting of data with materials-specific machine learning models. Here, we present two projects, the Materials Data Facility (MDF) and the Data and Learning Hub for Science (DLHub), that address these needs. We use examples to show how MDF and DLHub capabilities can be leveraged to link data with machine learning models and how users can access those capabilities through web and programmatic interfaces.
1 Introduction
Materials science has growing data and software resources, but researchers still lack easy access to sufficiently broad, high-quality machine-readable data and struggle to navigate diverse tools. MDF and DLHub address these needs by connecting data sources, analysis tools, and machine-learning models through discoverable, reusable infrastructure.
- Motivation: Researchers lack easy access to high-quality machine-readable data of sufficient volume and breadth for many materials-science problems.
- Motivation: Applying machine learning to materials discovery requires approaches that help researchers discover, access, integrate, apply, and share diverse data and software.
- Contribution: The article presents the Materials Data Facility (MDF) and Data and Learning Hub for Science (DLHub) as infrastructure systems addressing these ecosystem needs.
- MDF: MDF connects materials-data producers and consumers by collecting data from many sources, enriching it with tools such as metadata extraction and quality control, and routing it to multiple destinations.
- DLHub: DLHub enables researchers to describe, publish, discover, and run machine-learning models and related data-transformation tools, including complete publishable pipelines.
- Article scope: The article reviews the materials-data ecosystem, describes MDF and DLHub architectures and features, and presents three materials-science applications.
2 The Materials Data Ecosystem
The materials-data ecosystem includes curated repositories, general-purpose publication services, curation and provenance tools, and software for materials informatics. Curated databases are immediately usable by analysis software but represent only a small fraction of useful materials data.
- Data resources: Materials researchers can draw on specialized databases, general-purpose repositories, curation systems, provenance tools, and materials-informatics software.
- Data resources: Curated databases provide well-structured data that is immediately accessible to analysis software, but they represent only a small fraction of useful materials data.
- Curation and provenance: LIMS and workflow-management tools such as 4CeeD and Materials Commons support data curation and provenance tracking as data are produced.
- Software: Materials-informatics tools include MAST-ML, pyMKS, matminer, pymatgen, and ASE, which simplify or automate common pipeline tasks.
- Registries: The NIST Materials Resource Registry is a federated set of registries for discovering datasets, software, projects, and organizations relevant to materials science.
3 Materials Data Facility (MDF)
MDF is an interconnection point for publishing, enriching, discovering, and retrieving materials data across independent sources. Its modular services automate data sharing and materials-aware processing while supporting unified search across heterogeneous repositories.
- Purpose: MDF addresses fragmented materials-data workflows by allowing producers to dispatch results broadly and consumers to discover and aggregate data from independent sources.
- Purpose: MDF supports publication regardless of data size, type, and location, automated metadata extraction, and unified search across sources with different vocabularies and schemas.
- Scale: MDF currently stores 30 TB of simulation and experimental data and indexes hundreds of external datasets, producing millions of metadata records.
- Architecture: MDF comprises MDF Publish, a decentralized dataset repository; MDF Discover, a cloud-hosted search index; and MDF Connect, which links MDF and external services.
- MDF Connect: MDF Connect collects data from many locations, enriches and transforms it using materials-aware processes, and dispatches data and metadata to selected community services.
- MDF Connect: MDF Connect extracts general and scientific information, merges it with user metadata into MDF records, and can dispatch the resulting information to discovery and curation services.
- MDF Discover: MDF Discover supports simple and advanced search through full-text, field, range, fuzzy, wildcard, and faceting operations.
- Access: MDF Forge provides Python-based programmatic searches and retrieval, complementing web access for users working with diverse data.
4 The Data and Learning Hub for Science (DLHub)
DLHub addresses the difficulty of sharing and reusing machine-learning models by making models and associated tools describable, discoverable, publishable, and remotely executable. Its hosted servables reduce installation overhead and connect model capabilities with accessible metadata and interfaces.
- Motivation: Researchers face inefficiencies because materials machine-learning models and related codes are difficult to install, train, run, and share.
- Capabilities: DLHub provides facilities for describing, publishing, discovering, and running machine-learning models and associated data-transformation and analysis codes.
- Capabilities: DLHub lets researchers discover models through metadata and invoke them remotely on supplied inputs without downloading or configuring them locally.
- Implementation: DLHub reduces reuse overhead by automatically constructing portable servables, registering their metadata, and deploying them on scalable computing systems.
- Workflow: DLHub supports model description, publication, discovery, and execution through its SDK, REST API, and command-line interface.
- Workflow: Descriptive metadata such as authors, source-code links, input types and shapes, and application domain help users assess whether a servable fits their problem.
- Access control: Users can set separate access policies for servable metadata and the servable itself, allowing discovery while restricting execution.
- Execution: DLHub supports synchronous and asynchronous execution modes, enabling servables with both fast and slow execution times.
5 Science Use Cases
The three use cases connect MDF-hosted or indexed data with DLHub machine-learning servables for band-gap prediction, structure prediction, and high-quality atomization-energy estimation. Together, they demonstrate data querying, model publication, lightweight execution, and scalable prediction workflows.
- Overview: Three use cases link MDF data with DLHub servables to predict band gaps, material structures, and atomization energies.The workflows combine hosted or indexed datasets with models exposed through DLHub.
- Band-gap prediction: MDF metadata records describe 180,902 optical images and spectra, enabling training and test datasets to be selected by index or composition.Partitioning the former single HDF5 dataset through user queries enables new applications and data-mixing opportunities.
- Band-gap prediction: Published DLHub models extend prior optical-image work with VAE, autoencoder, and color-clustering approaches for band-gap prediction.The workflow retrieves image and spectrum data through MDF, runs models through DLHub, and examines image reconstructions and latent-space relationships with color and band gap.
- Coherent diffraction imaging prediction: The CDI use case publishes a GitHub model as a DLHub servable that predicts material structure from X-ray diffraction data.Researchers can run the servable with three lines of code after data-retrieval code, with minimal software-installation overhead.
- Molecular atomization energies: The molecular-energy use case uses Δ-learning models to estimate G4MP2-level atomization energies from faster B3LYP calculations, achieving around 10 meV accuracy.Through DLHub, the model can correct B3LYP energies for hundreds of molecules per second, with throughput scalable by changing servable replicas.
6 Conclusion
MDF and DLHub provide cohesive infrastructure services for automating data sharing, enriching and discovering materials data, and applying curated machine-learning models. The examples show how these capabilities can streamline materials discovery while remaining extensible through distributed execution, linked servables, and future connections to other data services.
- 6 Conclusion: MDF and DLHub automate data sharing among heterogeneous services, enrich data with general and materials-specific metadata, and simplify discovery, aggregation, and reuse.Their capabilities are provided as services, reducing user effort and allowing other data services to leverage them modularly.
- 6 Conclusion: The systems provide a curated library of machine-learning models and processing logic that can be applied to new data streams.This connects data services with reusable materials-specific learning capabilities.
- 6 Conclusion: Examples demonstrate that cohesive MDF and DLHub infrastructure can streamline complex materials discovery and deliver machine-learning study results to a broad audience.The article presents examples showing how these capabilities support materials-science applications.
- 6 Conclusion: Future work will connect MDF with other data services and automate metadata extraction across hundreds of common materials-science file types.The authors note that metadata-extraction work is currently fragmented across several code bases.
- 6 Conclusion: Although repositories increasingly support open automated harvesting through REST APIs, the lack of a shared authentication strategy remains a challenge.
- 6 Conclusion: DLHub is intended to support model execution on distributed resources and linking servables to represent complex machine-learning application logic.The planned resources include Jetstream, Amazon Web Services, and DOE Leadership Computing Facilities.
Code and Data Availability
Because MDF and DLHub are evolving, their code will change over time; the authors provide worked examples and public access to related code, packages, documentation, and interfaces.
- The authors provide a growing repository of worked examples from various scientific domains to support reproduction and extension of the work.These examples are accessible via the DLHub website.
- MDF-related code, packages, documentation, and interfaces are available through the Materials Data Facility website.
- DLHub-related code, including its SDK and CLI, documentation, and interfaces are available through the DLHub website.