Source-linked AI summary
Methods for comparing uncertainty quantifications for material property predictions
Kevin Tran, Willie Neiswanger, Junwoong Yoon, Qingyang Zhang, Eric Xing, Zachary W. Ulissi
TL;DR
Catalysis and materials-informatics workflows need reliable uncertainty estimates, yet the field lacks a standard way to compare them. The paper proposes figures and metrics for evaluating UQ and applies them to DFT adsorption-energy prediction, finding the best performance from a convolutional-neural-network/Gaussian-process model while noting implementation overhead as a limitation.
Problem
The computational catalysis field has few comparisons and no standard procedure for judging the quality of uncertainty estimates used with predictive models.
Method
The study presents a protocol with figures and performance metrics for comparing UQ methods and applies it to DFT-calculated adsorption-energy prediction.
Results
The best-performing method was a model combining convolutional-neural-network features with a Gaussian process for adsorption-energy prediction and UQ.
Takeaways & Limitations
The protocol provides a way to visualize and quantify UQ characteristics for objective comparison of methods.
Takeaways & Limitations
The neural-network implementation may impose enough overhead that a Bayesian neural network is not worth the investment.
Abstract
from arXiv · showhide
Data science and informatics tools have been proliferating recently within the computational materials science and catalysis fields. This proliferation has spurned the creation of various frameworks for automated materials screening, discovery, and design. Underpinning these frameworks are surrogate models with uncertainty estimates on their predictions. These uncertainty estimates are instrumental for determining which materials to screen next, but the computational catalysis field does not yet have a standard procedure for judging the quality of such uncertainty estimates. Here we present a suite of figures and performance metrics derived from the machine learning community that can be used to judge the quality of such uncertainty estimates. This suite probes the accuracy, calibration, and sharpness of a model quantitatively. We then show a case study where we judge various methods for predicting density-functional-theory-calculated adsorption energies. Of the methods studied here, we find that the best performer is a model where a convolutional neural network is used to supply features to a Gaussian process regressor, which then makes predictions of adsorption energies along with corresponding uncertainty estimates.
Introduction
Materials and catalysis increasingly rely on surrogate models with uncertainty estimates, but the field has few comparisons or standard procedures for judging their quality. This study introduces a comparison protocol and illustrates it on DFT-calculated adsorption-energy models.
- Motivation: Frameworks for materials screening, discovery, and design increasingly use machine-learning models with uncertainty quantification.These frameworks include active learning, Bayesian optimization, active search, and goal-oriented design.
- Motivation: UQ can indicate when predictions should be trusted and support online, active materials-discovery frameworks.The passages connect UQ with selecting or prioritizing subsequent computational work.
- Motivation: Uncertainty quantification aims to estimate the likelihood of outcomes associated with a predicted quantity.It may return confidence intervals or probability distributions over possible outcomes.
- Study objective: Few studies have compared UQ methods in catalysis and materials informatics, motivating a protocol for objective comparison.The protocol is applied to models predicting DFT-calculated adsorption energies.
- Study objective: The study also reports anecdotal case-study insights while warning that they may not transfer to other applications.The authors share them to help others build intuition.
Methods
The study evaluates UQ methods on a large DFT adsorption-energy dataset using consistent predictive uncertainty outputs and application-dependent data splitting. It compares neural, Bayesian, dropout, ensemble, and Gaussian-process approaches while introducing a units-matched sharpness formulation.
- Dataset: The dataset contains adsorption energies for H, CO, OH, O, and N adsorbates, with counts reported separately for each species.The supplied passages report 21,269 H, 18,437 CO, 3,464 OH, 2,515 O, and 1,594 N adsorption energies.
- Dataset: DFT calculations used VASP 5.4 with rPBE functionals, specified k-point grids and energy cutoffs, slab replication, and vacuum spacing.The setup fixed bottom slab layers and omitted spin magnetism and dispersion corrections.
- Regression methods: The study compares regression procedures that return predictive distributions, enabling common uncertainty-performance metrics.Methods include crystal-graph neural networks, ensembles, Bayesian neural networks, dropout neural networks, and Gaussian processes.
- Regression methods: For the neural-network ensemble, the prediction is the mean across five models and uncertainty is their prediction standard deviation.The five models use subsampled training data and independently randomized initial weights.
- Regression methods: The Bayesian neural network approximates parameter posteriors with variational inference, sampling parameters 20 times to obtain predictions and uncertainty.The final prediction is the mean of sampled predictions and uncertainty is their standard deviation.
- Performance metrics: The proposed sharpness formulation adds a square-root operation so its value has the same units as the predictions.The formulation is presented as a new version of sharpness.
Results
The results evaluate uncertainty quantification methods using accuracy, calibration, and sharpness, with additional comparisons based on negative log-likelihood. The methods show similar inlier accuracy but differ substantially in calibration, sharpness, and NLL, favoring CFGP overall.
- All methods have virtually identical MDAE results, while their MAE results are within 10% of each other, suggesting comparable predictive accuracies for inliers.
- The plain GP has the highest RMSE among the methods, indicating the worst predictive accuracy for outliers.
- The NN ensemble, BNN, and Dropout NN are overconfident, whereas GP is underconfident and NN∆NN, GPNN−µ, and CFGP are relatively calibrated.The more calibrated methods all first use a neural network for prediction and then an in-series method to estimate uncertainty.
- The NN ensemble, BNN, and Dropout NN produce the sharpest uncertainties, but at the cost of calibration.Among the more calibrated models, NN∆NN has sharpness 0.16 eV, GPNN−µ 0.21 eV, and CFGP 0.24 eV.
- CFGP achieves the best NLL, approximately 2,800, while GP and GPNN−µ have moderate NLL values of approximately 6,000.The underconfident GP has better NLL than NN∆NN despite worse miscalibration area, calibration error, and sharpness; the three most overconfident models have the worst NLL.
- The authors select CFGP because it offers competitive accuracy, calibration, and sharpness while requiring less information than GPNN−µ.The broader evaluation combines figures and metrics for accuracy, calibration, sharpness, and NLL.
Conclusions
The paper presents a broadly applicable procedure for holistic comparison of uncertainty-quantification methods, assessing accuracy, calibration, and sharpness through coordinated metrics and figures. In a DFT adsorption-energy case study, the Convolution-Fed Gaussian Process performed best, while GP-based and in-series methods showed more conservative and calibrated uncertainty estimates, respectively.
- Comparison procedure: The procedure combines error metrics, calibration curves, sharpness plots, and negative log-likelihood to judge UQ methods holistically.NLL assesses accuracy, calibration, and sharpness simultaneously, while the full metric-and-figure ensemble supports relative comparisons.
- Case study: Six methods were tested for predicting DFT-calculated adsorption energies with uncertainty quantification.
- Case study: The Convolution-Fed Gaussian Process used pre-trained convolutional-neural-network outputs as features for a Gaussian process making probabilistic predictions.This method was identified as the best performer in the case study.
- Case-study findings: GP-based methods tended to produce higher and more conservative uncertainty estimates than methods using only neural networks and neural-network derivatives.
- Case-study findings: In-series methods tended to be more calibrated than methods predicting values and uncertainties simultaneously.In-series methods first make value predictions and then use a subsequent model to estimate uncertainty.
- Scope and future work: These case-study results are limited to the dataset, and outcomes may vary across applications, models, or hyperparameters.The comparison procedure itself is described as broadly applicable despite this scope boundary.
- Scope and future work: Future work should consider recalibration when feasible UQ methods provide insufficiently calibrated uncertainty predictions.Recalibration was omitted here to simplify the illustration, assess innate method performance, and reduce overhead.
- Scope and future work: UQs may incorporate prior knowledge rather than being derived strictly from data, including inductive distinctions between optimized and unoptimized structures.
Code availability
The paper provides code for the results discussed, with dependencies listed in the repository.
- Code for reproducing the paper’s results is available in the referenced GitHub repository.The repository also lists its code dependencies.