Source-linked AI summary
A critical examination of compound stability predictions from machine-learned formation energies
Christopher J. Bartel, Amalie Trewartha, Qi Wang, Alexander Dunn, Anubhav Jain, Gerbrand Ceder
TL;DR
Machine learning is attractive for discovering new materials because materials prediction suits high-throughput techniques. This work evaluates learned formation energies against DFT-related stability behavior and finds that they do not reproduce DFT-calculated relative behavior reliably enough to match DFT’s usefulness for discovery.
Problem
Machine learning is being explored for discovering new materials, and materials prediction is a natural fit for high-throughput techniques.
Method
The work evaluates compositional and structural machine-learning approaches, including a structural model based on the CGCNN framework, across Materials Project compounds.
Results
Learned formation energies do not reproduce DFT-calculated relative behavior reliably, while many models predict compounds as stable that DFT finds unstable, impeding efficient materials discovery.
Takeaways & Limitations
More critical evaluation is needed before treating machine-learning models as comparably useful to DFT for discovering new materials.
Takeaways & Limitations
The stable quaternary compounds in the Li-Mn-TM-O chemical space have not yet been fully explored in the Materials Project database.
Abstract
from arXiv · showhide
Machine learning has emerged as a novel tool for the efficient prediction of materials properties, and claims have been made that machine-learned models for the formation energy of compounds can approach the accuracy of Density Functional Theory (DFT). The models tested in this work include five recently published compositional models, a baseline model using stoichiometry alone, and a structural model. By testing seven machine learning models for formation energy on stability predictions using the Materials Project database of DFT calculations for 85,014 unique chemical compositions, we show that while formation energies can indeed be predicted well, all compositional models perform poorly on predicting the stability of compounds, making them considerably less useful than DFT for the discovery and design of new solids. Most critically, in sparse chemical spaces where few stoichiometries have stable compounds, only the structural model is capable of efficiently detecting which materials are stable. The non-incremental improvement of structural models compared with compositional models is noteworthy and encourages the use of structural models for materials discovery, with the constraint that for any new composition, the ground-state structure is not known a priori. This work demonstrates that accurate predictions of formation energy do not imply accurate predictions of stability, emphasizing the importance of assessing model performance on stability predictions, for which we provide a set of publicly available tests.
Introduction
Machine-learned formation energies can be accurate, but that accuracy does not reliably translate into stability predictions. Across compositional models, relative stability remains difficult to resolve, especially in sparse chemical spaces, motivating structural models and stability-focused evaluation.
- Learning formation energy from chemical composition: ΔHd spans a much smaller energy range than ΔHf, making stability a more sensitive prediction task.Materials Project values span -1.42 ± 0.95 eV/atom for ΔHf and 0.06 ± 0.12 eV/atom for ΔHd.
- Learning formation energy from chemical composition: Formation enthalpy predictions substantially improve over the stoichiometry-only ElFrac baseline, reducing MAE by 27-74%.For most models, the MAE is comparable to the expected numerical disagreement between Materials Project and experiment.
- Implicit stability predictions from learned formation enthalpies: Despite similar MAE, compositional models show little error cancellation for ΔHd and generally cannot distinguish stable from unstable compounds within a chemical space.This contrasts with DFT, whose errors systematically cancel for stability predictions.
- Implicit stability predictions from learned formation enthalpies: The study concludes that improving elemental attributes alone is unlikely to transform stability prediction, whereas qualitative architectural changes such as Roost may help.The models remain limited for efficiently accelerating discovery because many predicted-stable materials are false positives.
Methods
The study evaluates formation-energy and stability prediction using Materials Project data, cross-validation, convex-hull analysis, and compositional and structural machine-learning models. Models are trained and tested through several hold-out procedures, including a targeted Li-Mn-TM-O chemical-space test.
- Scope and limitations: The dataset was queried on July 26, 2019, and its calculated decomposition energies may change as new stable materials are added to Materials Project.This evolving-database limitation affects training and testing values over time.
- Materials Project data: 85,014 unique non-elemental chemical formulas were extracted from Materials Project, using the most negative formation energy per formula as the ground state.Decomposition energies were obtained through convex-hull construction, with Materials Project corrections applied to thermodynamic properties.
- Evaluation workflow: Five-fold cross-validation generated held-out formation-energy predictions, which were then propagated through convex-hull analysis to obtain predicted decomposition energies and stability classifications.The same workflow was applied to a Li-Mn-TM-O test set excluded from training.
- Evaluation workflow: The Li-Mn-TM-O test set contained 267 quaternary compounds withheld while models were trained on the remaining Materials Project entries.An analogous procedure was used when models were trained directly on decomposition energy.
- Machine-learning models: Compositional representations included ElFrac, Meredig, Magpie, AutoMat, ElemNet, and Roost, with XGBoost used for several models and neural-network training specified for ElemNet and Roost.Roost was excluded from the classification analysis because adapting it for classification required more extensive changes.
- Structural model: CGCNN was trained with nested five-fold cross-validation, using training, validation, and test partitions and selecting models by validation mean absolute error.For Li-Mn-TM-O, five selected models produced averaged predictions for the 267 test compounds.
Supplementary Information
The supplementary information provides additional ROC curves, retraining and learning-curve analyses, alternative decomposition-energy evaluations, and performance tables. These materials extend the main stability assessment across thresholds, training targets, and computational costs.
- Supplementary figures: ROC curves evaluate models trained on formation energy across stability thresholds, with true- and false-positive rates defining classification performance.Changing the threshold changes the stable/unstable classification analysis, not the formation-energy model itself.
- Supplementary figures: Figure S2 retrains models after excluding 267 Li-Mn-TM-O quaternary compounds and compares predicted and Materials Project formation energies for those compounds.
- Supplementary figures: Figures S3 and S4 reproduce decomposition-energy and stability analyses using models trained on decomposition energy instead of formation energy.
- Supplementary figures: Figure S5 reports compositional-model formation-energy learning curves, averaging mean absolute error over five random train/test splits with standard-deviation error bars.The final point uses five-fold cross-validation at 68,011 training examples.
- Supplementary tables: Table S1 reproduces the targeted Li-Mn-TM-O evaluation for models trained on decomposition energy, while Table S2 reports compositional classification performance.Table S2 defines stable compounds as ΔHd ≤ 0 and excludes Roost from classification.
- Supplementary tables: Table S3 reports training and inference times for predicting formation energy on 68,011 training and 17,013 inference compounds.AutoMat training time is specified by the user.