Fast and Accurate Uncertainty Estimation in Chemical Machine Learning

Felix Musil; Michael J. Willatt; Mikhail A. Langovoy; Michele Ceriotti

Source-linked AI summary

Fast and Accurate Uncertainty Estimation in Chemical Machine Learning

Felix Musil, Michael J. Willatt, Mikhail A. Langovoy, Michele Ceriotti

arXiv:1809.07653v1Sep 20, 2018physics.chem-ph

TL;DR

Machine-learning predictions need reliable uncertainty estimates alongside high accuracy. The paper presents a resampling-based estimator and reports reliable estimates for small organic molecules, while noting that evaluation may cost more for some learning schemes.

Problem
Machine-learning models seek high-accuracy predictions while also needing estimates of the uncertainty associated with particular inputs.
Method
The method uses multiple resampling models, with each model producing a prediction that contributes to the uncertainty estimate.
Results
The sub-sampling estimator was reliable for small organic molecules, supported by log-likelihood results and agreement between true and predicted quantities.
Takeaways & Limitations
Accurate and inexpensive uncertainty estimation can support more reliable data-driven predictions.
Takeaways & Limitations
For some learning schemes, including neural networks, evaluating uncertainty may have substantial cost.

Abstract

from arXiv · show

We present a scheme to obtain an inexpensive and reliable estimate of the uncertainty associated with the predictions of a machine-learning model of atomic and molecular properties. The scheme is based on resampling, with multiple models being generated based on sub-sampling of the same training data. The accuracy of the uncertainty prediction can be benchmarked by maximum likelihood estimation, which can also be used to correct for correlations between resampled models, and to improve the performance of the uncertainty estimation by a cross-validation procedure. In the case of sparse Gaussian Process Regression models, this resampled estimator can be evaluated at negligible cost. We demonstrate the reliability of these estimates for the prediction of molecular energetics, and for the estimation of nuclear chemical shieldings in molecular crystals. Extension to estimate the uncertainty in energy differences, forces, or other correlated predictions is straightforward. This method can be easily applied to other machine learning schemes, and will be beneficial to make data-driven predictions more reliable, and to facilitate training-set optimization and active-learning strategies.

1 Introduction

Machine-learning models can achieve high accuracy for atomistic properties at much lower computational cost than electronic-structure calculations, but aggregate error scores do not reveal uncertainty for individual predictions. The paper therefore examines resampling-based uncertainty estimates for identifying less reliable inputs and guiding training-set construction.

Accurate machine-learning predictions can approach the underlying electronic-structure method while costing many orders of magnitude less.This cost advantage enables investigations that would otherwise be prohibitively expensive.
Individual prediction uncertainty is needed because RMSE and MAE summarize average residual magnitude rather than the error of a particular input.Such estimates can indicate whether a model is interpolating or extrapolating.
Prediction uncertainty can support more confident scientific conclusions and highlight underrepresented regions for training-set expansion.
Resampling trains models on different portions of the training set and uses the resulting prediction distribution to estimate uncertainty.The paper compares this estimator with GPR's predictive standard deviation and evaluates estimators using likelihood and cross-validation.
The study uses SOAP-GPR to assess uncertainty for QM9 formation energies and 1H NMR chemical shieldings in molecular crystals.

2 Methods

The paper first reviews GPR, projected-process approximations, and uncertainty estimators before developing resampling estimators and maximum-likelihood criteria for comparing and optimizing them.

The methods section introduces GPR, projected-process approximations, and associated uncertainty estimators as the paper's foundational framework.
It then develops resampling estimators of sample statistics and uses maximum likelihood to assess and optimize statistical estimators.

2.1 Gaussian Process Regression

GPR models predictions from observed input-property pairs using a kernel-based Gaussian-process prior, while predictive variance supplies an uncertainty estimate. A projected-process approximation compresses the model through an active representative subset, but exact GPR becomes prohibitive for large training sets.

The Gaussian-process assumption makes the prediction a normal distribution whose standard deviation estimates uncertainty for each input.
GPR predicts a target property from observed input-property pairs using a kernel that represents similarity between inputs.
Exact GPR becomes computationally prohibitive for large training sets, motivating sparse approximations such as the projected process.
The projected-process approximation selects an active subset as a new regression basis and augments it through correlations with the full training set.
Under the model assumption, the interval [y(X)−σPP(X), y(X)+σPP(X)] contains the target 67% of the time.

2.2 Resampling

Resampling estimates prediction uncertainty from an ensemble of models trained on random subsets of the training data, providing empirical distributions and correlated multi-input uncertainty estimates. The paper calibrates sub-sample-based estimates with transformations and maximum likelihood, finding no uncertainty-estimation advantage for bootstrapping over sub-sampling in the studied sparse GPR models.

Resampling trains independent models on random data subsets, using their prediction ensemble as a non-parametric uncertainty distribution.
The prediction ensemble can characterize non-Gaussian error statistics and jointly estimate uncertainty for correlated combinations of multiple inputs.
Sub-sampling draws smaller samples from the correct distribution, whereas bootstrapping draws full-size samples from the wrong distribution.
Maximum-likelihood-fitted linear or nonlinear transformations extrapolate sub-sampling uncertainties toward the full-training-set limit without assuming a particular functional form.
Bootstrapping offers no advantage over sub-sampling for uncertainty estimation with the studied GPR projected-process models, so the paper focuses on the simpler, cheaper method.
With sparse GPR and a shared representative set, evaluating uncertainty requires one kernel computation plus NR scalar products, making the added prediction cost effectively negligible.

2.3 Log-likelihood assessment of uncertainty estimates

The section introduces log-likelihood as a way to assess uncertainty estimates alongside prediction accuracy. Under an uncorrelated Gaussian assumption, the best uncertainty model gives the highest likelihood on a common test set.

Log-likelihood evaluates whether predicted means and uncertainties assign high probability to observed test targets.The targets are modeled as normally distributed with model-determined means and variances.
The likelihood sums log P(y_n|X_n) across test points, treating the targets as uncorrelated Gaussian observations.The uncorrelated-target assumption makes the total log-likelihood additive across test inputs.
For models sharing a test set, the highest log-likelihood identifies the better uncertainty estimate.This comparison applies when the models differ only in their uncertainty estimates.

mates

The method calibrates uncertainty estimates with maximum likelihood and can exploit resampling-based validation. It also addresses broad or correlated resampling distributions, while noting limits on variance scaling and overfitting control.

Resampling distributions can be corrected for artificial correlations, and broader transformations can replace simple variance scaling.The same framework is applied to GPR variances and bootstrapping, while transformations may correct overly broad or narrow estimates.
Maximum likelihood determines a variance scale factor on validation data, which is then applied when testing.The factor is optimized from validation residuals and used to scale GPR variances or resampling distributions.
Leave-one-out predictions from absent training inputs provide an internal validation procedure for estimating the scale factor.Inputs contribute when they are absent from enough resampling datasets, producing an effective validation set without explicitly constructing one.
For sub-sampling, the expected absent-input fraction is 1 − n/N, so n = N/2 yields roughly half the training set as effective validation data.With NR = 10, slightly more than 50% of inputs are expected to be absent from at least five models.
The internal validation procedure cannot be used to scale GPR variances, and cross-validation only mitigates overfitting risk.The authors also note that model-complexity penalties are customary in maximum-likelihood estimation.

2.5 Smooth Overlap of Atomic Positions

The paper uses SOAP to compare atomic environments and structures while preserving invariance to rotations, translations, and identical-atom permutations. Kernel treatment differs between atom-level shieldings and molecule-level formation energies.

The SOAP kernel is invariant to rotations, translations, and permutations of identical atoms.
SOAP represents atomic environments through power-spectrum inner products built from local neighbor densities.The density uses atom-centred Gaussians expanded in radial functions and spherical harmonics, with a cutoff restricting nearby atoms.
The environmental kernel is used directly for chemical shieldings but averaged over environments for molecular formation energies.This reflects whether the target property belongs to an individual atom or an entire molecule.
SOAP kernels are dimensionless, so property-unit scaling is required for a dimensionally meaningful GPR uncertainty.The regularization parameter is scaled consistently, without changing the prediction itself.

2.6 Benchmark Datasets

The benchmarks cover molecular formation energies and hydrogen chemical shieldings using QM9 and CSD-derived datasets. Both workflows use fixed train/validation/test partitions, representative subsets, and repeated random sub-sampling.

CSD-derived datasets: The CSD-500 structures were rattled to create the CSD-890-R benchmark with off-equilibrium environments at 0.25 Å and 0.5 Å RMSD.The source CSD-500 set contains 500 optimized crystal structures, while the resulting benchmark contains 890 structures.
Reference calculations: The electronic-structure reference calculations used Quantum Espresso with PBE ultrasoft pseudopotentials, GIPAW, and specified plane-wave, charge-density, convergence, and k-point settings.Scalar chemical shieldings were obtained from the diagonal average of the shielding tensor.
CSD-derived datasets: For CSD hydrogen environments, 30k examples were split into 20k training, 5k validation, and 5k test environments.The 20k training environments were FPS-sorted, and the 10k most diverse were used as the PP representative set.
Resampling design: Sixteen random training-set sub-samples were selected for each sub-sample size, with likelihood calculations counting inputs absent from at least five models.
QM9: For QM9, 30k structures were split into 20k training, 5k validation, and 5k test structures, with formation energies including zero-point corrections as targets.The PP representative set used the 5k most distant environments among 361k training environments.

3 Results and Discussion

Maximum-likelihood scaling stabilizes resampling-based uncertainty estimates across sub-sample sizes, while nonlinear scaling improves agreement between predicted and actual errors across shielding and formation-energy tasks.

Overall uncertainty-estimation results: After scaling, final log likelihood becomes insensitive to sub-sample size, while nonlinear scaling further improves the uncertainty model.Table 1 reports this pattern for the shielding experiments, and Table 2 reports the same trend for QM9 formation energies.
CSD 1H NMR chemical shieldings: The 5k-environment estimator strikes the best balance before scaling, whereas rescaling substantially stabilizes log likelihood across resampling estimators.The 5k subset is one quarter of the training set and balances distributional representativeness against subset size.
CSD 1H NMR chemical shieldings: After rescaling, the GPR-PP estimator achieves noticeably higher log likelihood and the best agreement between predicted and actual error distributions.The GPR-PP result is reported for the shielding analysis; large-sample resampling estimators instead span nearly two orders of magnitude in predicted uncertainty.
CSD 1H NMR chemical shieldings: Cross-validated nonlinear scaling improves agreement between resampling and actual error distributions and brings their log likelihood close to the GPR-PP level.The transformation rescales σ(X_n) as α σ(X_n)^γ, with α and γ optimized by cross-validation.
QM9 formation energies: For QM9 formation energies, the same trend recurs, while scaled GPR and sub-sample estimators become more similarly reliable across the two property types.The study contrasts global formation energies with local chemical shieldings and reports broadly comparable reliability after nonlinear scaling.

4 Conclusions

The paper presents an inexpensive, reliable uncertainty estimator based on sub-sampling and sparse Gaussian Process Regression, validated for molecular energies and NMR chemical shieldings. It improves on standard GPR uncertainty estimation in computational scaling and supports broader uncertainty-aware modeling strategies.

The method provides an inexpensive uncertainty estimate for atomic and molecular machine-learning predictions using sub-sampling and sparse GPR.The approach is presented as a scheme for uncertainty estimation based on sub-sampling, with sparse GPR enabling low-cost evaluation.
The sub-sampling estimator was reliable for formation energies of small organic molecules and ^1H NMR chemical shieldings in organic crystals.Reliability was assessed using log likelihoods and agreement between true and predicted test-set error distributions.
The estimator improves on standard GPR uncertainty estimation in computational scaling while retaining reliable uncertainty predictions.The conclusions explicitly compare the method with the standard GPR uncertainty estimate from the perspective of computational scaling.
Resampled models remain accurate when heavily compressed, supporting model compression for faster training and inference.The paper identifies compression as useful because it is often required to accelerate model training and inference.
The objective uncertainty-optimization framework can also be applied to other machine-learning schemes, although uncertainty evaluation may be substantially more costly than for sparse GPR.Neural networks are given as an example of another applicable scheme, with a computational-cost qualification.
Ensemble predictions make uncertainty estimation straightforward for derived properties and can support training-set optimization, active learning, and uncertainty-based model combination.The paper also proposes combining models so predictions can degrade gently toward less accurate but more robust models in extrapolative regimes.