Source-linked AI summary
Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm
Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, Anubhav Jain
TL;DR
Materials machine-learning models lacked a standardized way to compare performance across datasets and tasks. The paper introduces Matbench and the automated Automatminer pipeline, finding that crystal graph networks perform better on larger tasks while conventional approaches remain competitive on smaller ones. The authors also identify a feature limitation in Automatminer’s inexpensive descriptor configuration.
Problem
Materials ML lacks a standard benchmarking method because studies often differ in datasets, cleaning, splitting, and generalization-error estimation.
Method
The paper introduces Matbench, a 13-task benchmark, and Automatminer, a fully automated pipeline for materials property prediction from materials primitives.
Results
Automatminer and crystal graph networks outperform or equal Random Forest on all but one task; graph networks are more accurate on most tasks with ~10^4 points or fewer?
Takeaways & Limitations
Matbench exposes algorithm-specific predictive strengths and supports consistent comparison of materials ML methods across task sizes.
Takeaways & Limitations
Automatminer’s inexpensive descriptor configuration omits features for determining 2nd-nearest-neighbor coordination.
Abstract
from arXiv · showhide
We present a benchmark test suite and an automated machine learning procedure for evaluating supervised machine learning (ML) models for predicting properties of inorganic bulk materials. The test suite, Matbench, is a set of 13 ML tasks that range in size from 312 to 132k samples and contain data from 10 density functional theory-derived and experimental sources. Tasks include predicting optical, thermal, electronic, thermodynamic, tensile, and elastic properties given a materials composition and/or crystal structure. The reference algorithm, Automatminer, is a highly-extensible, fully-automated ML pipeline for predicting materials properties from materials primitives (such as composition and crystal structure) without user intervention or hyperparameter tuning. We test Automatminer on the Matbench test suite and compare its predictive power with state-of-the-art crystal graph neural networks and a traditional descriptor-based Random Forest model. We find Automatminer achieves the best performance on 8 of 13 tasks in the benchmark. We also show our test suite is capable of exposing predictive advantages of each algorithm - namely, that crystal graph methods appear to outperform traditional machine learning methods given ~10^4 or greater data points. The pre-processed, ready-to-use Matbench tasks and the Automatminer source code are open source and available online (http://hackingmaterials.lbl.gov/automatminer/). We encourage evaluating new materials ML algorithms on the MatBench benchmark and comparing them against the latest version of Automatminer.
1. Lawrence Berkeley National Laboratory, Energy Technologies Area, 1 Cyclotron Road, Berkeley, CA 94720, United States
The paper addresses inconsistent comparison and manual model selection in materials machine learning by introducing Matbench and Automatminer. Matbench standardizes evaluation across diverse tasks, while Automatminer automates descriptor generation, preprocessing, feature reduction, and model selection.
- Motivation: Materials machine-learning studies often use different datasets, cleaning procedures, train/test splits, and error-estimation methods, complicating fair comparison.
- Motivation: Manual selection of descriptors and models is nontrivial across the broad range of materials tasks and can consume researcher time.
- Contribution: The benchmark combines a robust materials-ML test suite with an automatic reference model to reduce arbitrary model-selection effects and support comparisons across datasets.
- Matbench: Matbench contains 13 inorganic materials property-prediction tasks spanning small and large datasets, with consistent nested cross-validation for estimating regression and classification errors.
- Automatminer: Automatminer is a general-purpose, fully automated pipeline that accepts materials primitives and uses materials-specific featurization, feature reduction, preprocessing, and internal model selection.
- Evaluation: The authors evaluate Automatminer on Matbench against published machine-learning methods and show that the benchmark distinguishes predictive strengths and weaknesses among algorithms.
Results
Matbench provides 13 cleaned, diverse materials-property prediction tasks spanning multiple data sources, input types, properties, and dataset sizes. Automatminer is an automated pipeline that transforms compositions and/or crystal structures into predictions without researcher intervention, and its evaluation reveals performance differences across model families and dataset scales.
- Matbench test suite: 13 tasks span 10 datasets and cover experimental and computed materials properties across multiple application areas.The suite includes mechanical, elastic, electronic, optical, phonon, and thermodynamic properties for crystals, 2D materials, and disordered metals.
- Matbench test suite: 312 to 132,752 samples per task capture both scarce experimental properties and abundant computed properties.Task sizes vary across several orders of magnitude, including DFT-GGA formation energies.
- Matbench test suite: Precleaned datasets and consistent nested cross-validation support reproducible comparisons between models.The authors recommend using the tasks as-is to reduce model and sample selection biases.
- Automatminer pipeline: Automatminer accepts compositions and/or crystal structures, automatically featurizes and cleans inputs, reduces features, and searches internal AutoML pipelines.It produces predictions from a training dataset without further researcher intervention or hyperparameter tuning.
- Automatminer pipeline: Automatminer’s Express preset retains 95% or more of the accuracy of more expensive presets on multiple data-scarce tasks.The same preset fits all Matbench tasks without additional input or task-specific configuration.
- Benchmark results: Automatminer achieves the best nested-cross-validation score on 8 of 13 tasks and equals or outperforms Random Forest on all but formation-energy prediction.All algorithms outperform Dummy, whose errors are 68% to 299% higher than those of the best model for each task.
- Benchmark results: Graph networks improve faster with dataset size, while descriptor-based models perform better on small datasets and lose that advantage beyond 10^4 samples.Across six tasks with more than 10^4 samples, MEGNet and CGCNN scores improve relative to conventional methods; near 10^5 samples, the AutoML advantage is essentially lost.
- Benchmark results: Task details still matter: graph networks outperform traditional methods strongly on formation energies but less strongly on band-gap regression, while MEGNet decisively leads Automatminer on PhDOS.Among nine structure tasks, Automatminer and MEGNet each have the best score on four tasks, and CGCNN leads only on Materials Project band-gap regression.
Discussion
The discussion positions Matbench and Automatminer as extensible resources for consistent materials-ML evaluation and rapid pipeline development. Results also identify trade-offs: expensive presets yield diminishing returns, while model and feature coverage constrain performance and benchmark realism.
- Resources and evaluation: Automatminer is an extensible, universal platform intended to accelerate supervised-learning development and seed more specialized models.Its configurable stages and extensible featurizer design support adaptation beyond the demonstrated settings.
- Resources and evaluation: Matbench and Automatminer together provide a benchmark and reference pipeline for consistent, unbiased comparisons of materials property prediction algorithms.Matbench defines a consistent test procedure, while Automatminer supplies an automated baseline.
- Preset trade-offs: Minor performance improvements from more expensive presets require significant increases in computational time, producing diminishing returns.This trade-off is especially noticeable on small datasets, where many pipelines can instead be attempted within the time restriction.
- Preset trade-offs: 0.2% ROC-AUC improvement separates Express from Debug for experimental metallicity classification, while Heavy adds only another 0.6%.The reported averages are 0.919 for Express, 0.917 for Debug, and 0.925 for Heavy.
- Reference algorithm limitations: Automatminer lacks features for second-nearest-neighbor coordination and other medium-range structural information that may explain graph networks’ advantages on some thermodynamic tasks.Future development could incorporate graph-network embeddings or additional chemo-spatial descriptors because the pipeline is extensible.
- Reference algorithm limitations: Tree-ensemble methods generally outperform the other models searched by Automatminer, although this dominance partly reflects its limited model search space.The current search excludes nonlinear support-vector-machine kernels and neural networks, which may improve performance on large datasets.
Methods
Matbench datasets were curated from original or Matminer sources, cleaned into materials-input tasks, and evaluated with five-fold nested cross-validation. Descriptor-based Random Forest, Automatminer, and crystal graph models used specified featurization, optimization, and validation procedures.
- Dataset construction: Matbench v0.1 datasets were reconstructed from original or Matminer sources and curated to retain materials inputs and target variables while removing extraneous data.Additional filtering removed unphysical, uncommon, or unrepresentative samples and applied task-specific composition and structure criteria.
- Evaluation limitations: Nested cross-validation is computationally expensive and can produce high-variance error estimates across repeated runs, although it was judged adequate for Matbench model comparison.Alternative schemes may reduce computational cost or variability but rely on assumptions that do not hold for some models and noisy observations.
- Baseline models: Descriptor-based Random Forest and Automatminer used Matminer with identical data cleaning procedures and composition or structure featurizers.Random Forest used Magpie features for compositions and SineCoulombMatrix for structure tasks, while missing features were filtered or mean-imputed.
- Crystal graph models: Crystal graph models were trained and optimized by splitting the data according to the stated benchmark procedure, with regression or classification losses and early stopping based on validation performance.Training used mean squared error or binary cross-entropy, and validation used MAE or ROC-AUC, respectively.
Data Availability
The Matbench datasets, retrieval code, documentation, and Automatminer test code are openly available through online repositories and platforms. Readers are also encouraged to suggest modifications to the benchmark suite.
- Access: Instructions for downloading and using Matbench are available through the online Automatminer repository.The repository also contains the code for running the specific tests reported in the paper.
- Access: Matbench data can be interactively viewed on the Materials Project MPContribs-ML platform as serialized tabular data.Dataset retrieval and loading code is available in Matminer’s dataset_retrieval folder.
- Community and licensing: Readers are encouraged to suggest modifications to the Matbench dataset test suite through the Matminer help forum.Automatminer code is open source under a BSD-style license.
Competing Interests
The authors declare no competing interests.
- The authors declare no competing interests.
Supplementary Information
The supplementary information accompanies the paper titled “Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm,” authored by Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain.
- The paper is titled “Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm.”
- The authors are Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain.
1. Training and prediction timing
The timing comparison reports training and prediction costs for Automatminer, Random Forest, and graph networks across Matbench tasks using nested cross-validation.
- 13 Matbench tasks are shown for Automatminer and Random Forest, while MEGNet and CGCNN are evaluated only on 9 structure tasks.
- Each plotted point represents one nested-cross-validation fold for a task.
- Training and evaluation for Random Forest and Automatminer used a single 24-core CPU node with 64GB RAM.
- Linear regressions are provided on log-log timing plots for each algorithm.
2. Automatminer Configuration
Automatminer combines composition featurizers with configurable preprocessing and regression or classification models through a scikit-learn-compatible pipeline space.
- The Automatminer Express preset uses Matminer featurizers for the reported results.
- Composition representations include elemental fractions, stoichiometric norms, atomic-orbital estimates, and weighted elemental statistics or embeddings.
- The regression model space includes gradient boosting, decision trees, k-nearest neighbors, LassoLarsCV, linear SVR, random forests, ridge regression, and XGBoost.
- TPOT can stack preprocessing steps and machine-learning models into loosely structured tree-hierarchy pipelines.
- Hyperparameter grids are defined from model arguments, with parameter ranges specified in the Automatminer source code.
- The classification model space contains scikit-learn and XGBoost classifiers alongside preprocessing and feature-selection components.
3. Convolutional Graph Network Hyperparameters
The supplementary configuration documents CGCNN and MEGNet training settings and full network architectures, including layer connectivity, output shapes, and trainable-parameter counts.
- CGCNN training uses hyperparameters recommended in the CGCNN paper to generate the benchmark results.
- The CGCNN architecture breakdown enumerates each layer, trainable parameters, output shape, and connections to subsequent layers.
- The documented network configurations support analyzing neural-network behavior in relation to training-sample counts and overfitting risk.
- The MEGNet model has 98,185 trainable parameters.
- The CGCNN model architecture specifies 167,761 trainable parameters in total.
4. Automatminer Preset Comparison on Experimental
The preset comparison evaluates Debug, Express, and Heavy Automatminer configurations on experimental metallicity classification, relating computational intensity to predictive performance.
- For datasets smaller than 10^4 samples, Automatminer can require more than 1000x higher runtimes.
- Express and Heavy marginally improve ROC-AUC over Debug, but require much greater computational effort.
- The runtime gap between expensive and inexpensive pipelines may narrow as dataset size decreases.
- Debug, Express, and Heavy presets are ordered by increasing computational intensity in the metallicity-classification comparison.
- The evaluation uses mean values across folds from full 5-fold nested cross-validation, with error bars showing between-fold standard deviation.
- Debug retains more features because it performs only minimal correlation-based feature reduction.
5. Supplement References
This supplement lists references covering crystal graph networks, interpretable material-property prediction, Random Forests, materials data-mining tools, and automated pipeline optimization.
- Crystal graph and materials-property prediction references include Crystal Graph Convolutional Neural Networks, Graph Networks as a Universal Machine, and interpretable prediction of material properties.
- Machine-learning and materials-data infrastructure references include Random Forests, Matminer, pymatgen, scikit-learn, Keras, and NumPy.
- Additional references address automated tree-based pipeline optimization and machine-learning applications to formation energies, band gaps, atomization energies, and inorganic-material properties.
- The remaining references cover density-functional calculations, atomic electronegativities, electrostatic lattice potentials, and materials-science literature embeddings.