PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty Quantification Using Neural Networks for Molecular Property Prediction

Lior Hirschfeld, Kyle Swanson, Kevin Yang, Regina Barzilay, Connor W. Coley

arXiv:2005.10036v1cs.LGq-bio.QMstat.ML

TL;DR

The paper addresses uncertainty in molecular property prediction, especially for opaque neural networks. It finds no universally superior UQ method or consistently strong error ranking, while recommending practical techniques with important dataset- and split-level variation.

  • Problem

    Uncertainty is especially challenging to evaluate for neural networks because they are opaque and difficult to interpret.

  • Method

    The paper compares uncertainty-quantification approaches using complementary performance metrics, including negative log likelihood of observed errors.

  • Results

    No UQ method consistently produced strong error rankings or significantly outperformed all others across every performance metric.

  • Takeaways & Limitations

    The authors recommend the MPNN RF as a reasonable first approach while noting substantial variation across datasets and splits.

  • Takeaways & Limitations

    NLL can be uninformative under certain distance conditions, and method performance varies substantially across datasets and splits.

Abstract

from arXiv · show

Uncertainty quantification (UQ) is an important component of molecular property prediction, particularly for drug discovery applications where model predictions direct experimental design and where unanticipated imprecision wastes valuable time and resources. The need for UQ is especially acute for neural models, which are becoming increasingly standard yet are challenging to interpret. While several approaches to UQ have been proposed in the literature, there is no clear consensus on the comparative performance of these models. In this paper, we study this question in the context of regression tasks. We systematically evaluate several methods on five benchmark datasets using multiple complementary performance metrics. Our experiments show that none of the methods we tested is unequivocally superior to all others, and none produces a particularly reliable ranking of errors across multiple datasets. While we believe these results show that existing UQ methods are not sufficient for all common use-cases and demonstrate the benefits of further research, we conclude with a practical recommendation as to which existing techniques seem to perform well relative to others.

Introduction

The paper systematically evaluates uncertainty quantification methods for neural molecular-property regression across datasets, architectures, and complementary metrics. Results show substantial inconsistency, with no method reliably ranking errors or outperforming all others across tasks.

  • Motivation: Comparative performance is difficult to assess because datasets, evaluation criteria, and hyperparameter choices vary across studies.
  • Study design: The study evaluates UQ techniques on four public benchmark datasets and one synthetic dataset for regression tasks.
  • Study design: The methods are applied to MPNNs and FFNs with fingerprint representations to test whether UQ behavior remains constant across neural architectures.
  • Evaluation: The evaluation measures error-ranking ability, confidence-interval calibration, and whether quantitative uncertainty explains molecule-level prediction errors.
  • Results: No method consistently produces a strong ranking of errors on either MPNNs or FFNs, and performance is strongly dataset dependent.
  • Results: No UQ method significantly outperforms all others across performance metrics; methods vary across criteria within datasets and across datasets.
  • Implications: Existing UQ tools may isolate low-error predictions for some tasks, but all methods can perform poorly on other tasks whose identifying properties remain unknown.

Methods

The study compares four uncertainty-estimation strategies for regression models: ensemble-based, mean-variance, distance-based, and union-based methods. These methods differ in whether uncertainty comes from prediction variation, predicted variance, molecular proximity to training data, or a second model.

  • Four primary strategies are evaluated: ensemble-based, distance-based, mean-variance estimation, and union-based methods.
  • Distance-based methods use molecular distance from training examples as uncertainty, whereas union-based methods pass neural-network outputs to another uncertainty model.The paper distinguishes quantitative variance estimates from relative uncertainty indicators because their downstream uses differ.
  • Ensemble Based Methods: Ensemble methods train multiple models and use variation among their predictions as uncertainty.The ensemble prediction is the arithmetic mean of individual model outputs, while uncertainty is defined by their variance.
  • Ensemble Based Methods: Bootstrap, snapshot, and Monte Carlo dropout variants reduce or alter ensemble-training costs by changing data exposure, model snapshots, or stochastic prediction masks.Snapshot ensembles require one training run, while dropout uses one model with different masks during prediction.
  • Mean Variance Estimation (MVE): Mean-variance estimation modifies the network output to predict a Gaussian distribution whose variance serves as uncertainty.The approach is presented as an alternative to Bayesian neural networks, whose exact training is impractical for large datasets.

Evaluation Metrics

The evaluation uses complementary metrics that assess uncertainty-error ranking, calibration, and likelihood under predicted variances. It also accounts for whether an estimator is intended to represent a variance and tests performance across multiple molecular-property datasets and split types.

  • Ranking: Spearman’s rank correlation measures whether lower predicted uncertainty corresponds on average to lower absolute prediction error.A perfect rank correlation is not expected because random low-error outcomes can occur even for highly uncertain predictions.
  • Calibration: Calibration compares observed error fractions within predicted standard-deviation ranges against Gaussian expectations.The miscalibration area is the area between the observed and expected fraction curves.
  • Likelihood: Negative log likelihood evaluates observed errors under a zero-mean normal model whose variances are supplied by uncertainty estimates.The metric is averaged across molecules, and the study also examines the gap between achieved and minimum possible NLL.
  • Likelihood: Uncertainty estimates not intended as variances, such as distances, require calibration before NLL becomes informative.The study computes calibrated NLL by fitting aU(x) + b on validation data.
  • Experimental Design: The benchmark comprises four molecular-property datasets plus synthetic CLogP, using eight random splits and one scaffold split.The datasets include aqueous solubility, solvation energy, lipophilicity, atomization energy, and CLogP; scaffold splitting increases the likelihood of out-of-domain test molecules.

Results

Across five datasets and multiple metrics, UQ performance varied substantially by dataset, split, method, and evaluation criterion; no method was consistently superior. MPNN MVE, MPNN RF, and related RF/GP approaches often performed strongly, but calibration and error-ranking behavior remained inconsistent.

  • Cross-dataset variation: Benchmarking UQ on a single dataset could produce completely incorrect conclusions because performance varied strongly between datasets.Lipophilicity was difficult, whereas simulated CLogP supported relatively strong performance from many methods.
  • Spearman’s rank correlation coefficient: No method was consistently superior across datasets, and some methods occasionally showed negative Spearman correlations.MPNN MVE, MPNN RF, and FP RF were often highly ranked, but performance varied substantially.
  • Spearman’s rank correlation coefficient: Lipophilicity exposed a failure case: MPNN MVE produced higher-than-average error among its most confident predictions.This occurred despite more reliable error rankings on other datasets.
  • Miscalibration area: Miscalibration differed between split types: several MPNN methods had much larger areas on scaffold splits, while MPNN GP and MPNN RF often performed well there.Miscalibration area measures aggregate confidence alignment, not whether individual absolute errors are predicted well.
  • Negative log likelihood: For NLL, MPNN GP, MPNN RF, FP RF, and FP GP performed well, while MPNN Ensemble tended to underestimate error and CLogP scaffold splits challenged most estimators.Distance-based UQ metrics were excluded from quantitative NLL evaluation, and FFN MVE was excluded as an outlier.
  • Calibrated negative log likelihood: Calibration improved some methods on the first four datasets and reduced maximum observed NLL by nearly an order of magnitude, but sometimes harmed previously well-calibrated methods.The degradation was especially evident on CLogP and may reflect poor fits on particular random splits.

Discussion

The study compares UQ methods built on MPNN and FFN models with learned or fixed molecular representations using four complementary metrics. Although FFNs often had higher prediction RMSE than MPNNs, UQ methods showed similar relative behavior across architectures, while method rankings varied across datasets and metrics.

  • Evaluation framework: The evaluation uses Spearman correlation, miscalibration area, NLL, and calibrated NLL to assess complementary aspects of UQ quality.These metrics address error ranking, confidence-interval agreement, quantitative variance agreement, and linearly recalibrated variance agreement.
  • Architecture effects: Union-based methods outperformed virtually all ensembling methods on miscalibration area, despite substantial accuracy differences between MPNNs and FFNs.The authors note that architecture comparisons are complicated by these unequal model accuracies.
  • Architecture effects: FFN RMSE was often twice as high as MPNN RMSE on the same dataset, but both architectures usually achieved similar percentage RMSE reductions from UQ selection.Most UQ metrics were also similar across architectures, with greater variation for calibrated NLL.
  • Consistency across datasets: Method rankings were not consistent across datasets: some methods achieved standout performance on one dataset but only middling performance on another.This variation can make a method appear superior to baselines when evaluated on only one dataset or split.
  • Consistency across datasets: MPNN MVE isolated low-error predictions well on CLogP but poorly on lipophilicity: its top 25% had 75% lower RMSE on CLogP and higher RMSE on lipophilicity.This contrast illustrates how practical UQ utility can change sharply across datasets.
  • Overall method comparison: MPNN MVE and MPNN RF ranked best for uncertainty ordering, FP RF had the lowest overall miscalibration areas, and MPNN RF often produced the lowest NLL.MPNN GP, FP GP, and related RF methods were also competitive, with MPNN RF, MPNN GP, and MPNN MVE among the strongest neural-network methods overall.

Conclusion

The evaluation reveals substantial variation in uncertainty-estimation performance across datasets and splits, preventing a universally superior method. The authors recommend MPNN RF as a reasonable first approach while emphasizing this variability and identifying stacking as a promising direction.

  • Future directions: An ideal uncertainty estimator would show less performance variation between datasets than existing methods.The authors identify improved cross-dataset consistency as an important future goal.
  • Future directions: Stacking may improve consistency by combining multiple weak models into a meta-estimator.The paper notes that many method combinations and aggregation strategies remain unexplored.
  • Future directions: Future work could extend uncertainty quantification research to classification tasks.The paper specifically points to recent work in that area.
  • Practical recommendation: MPNN RF is recommended as a reasonable first approach, despite substantial variation across datasets and splits.The recommendation is made particularly in the context of the NLL evaluation metric.

Supporting Information Available

The paper makes its code, data, and tabulated results freely available, with supporting information containing additional evaluation results.

  • Availability: All code, data, and tabulated results used in the study are available online.The cited repository is github.com/lhirschfeld/ChempropUncertaintyQuantification.
  • Supporting Information: The Supporting Information contains additional results for the negative log likelihood evaluation.These materials extend the reported evaluation beyond the main text.
  • Supporting Information: The Supporting Information also contains results relating to the Wilcoxon signed-rank test..

Message Passing Networks

The paper’s authors are affiliated with institutions including MIT, the University of Cambridge, and the University of California Berkeley.

  • Affiliations: Lior Hirschfeld and Regina Barzilay are affiliated with MIT’s Computer Science and Artificial Intelligence Laboratory.The laboratory is located in Cambridge, Massachusetts.
  • Affiliations: Kyle Swanson is affiliated with the University of Cambridge’s Department of Pure Mathematics and Mathematical Statistics.The department is identified as being in Cambridge, UK.
  • Affiliations: Kevin Yang is affiliated with the University of California Berkeley’s Department of Electrical Engineering and Computer Sciences.The affiliation is in Berkeley, California.
  • Affiliations: Connor W. Coley is affiliated with MIT’s Department of Chemical Engineering.The department is located in Cambridge, Massachusetts.

Additional Results

The supplementary figures evaluate quantitative agreement between uncertainty estimates and observed absolute errors across five datasets. They compare raw and calibrated likelihood measures and summarize calibration-slope behavior across methods.

  • NLL-based evaluation: Figure S1 compares NLL Difference across UQ methods and five datasets, with lower values indicating better quantitative agreement.The panels cover Delaney aqueous solubility, freesolv solvation energy, lipophilicity, QM7 atomization energy, and CLogP heuristicly-calculated lipophilicity.
  • Calibrated evaluation: Figure S2 compares cNLL Difference across methods and datasets after allowing a linear adjustment to the uncertainty metric.Lower cNLL Difference indicates better agreement after calibration.
  • NLL-based evaluation: Figure S3 reports raw NLL for methods producing confidence intervals across all evaluated datasets.Distance-based metrics are not expected to produce meaningful quantitative results in this interpretation.
  • Calibrated evaluation: Figure S4 reports calibrated NLL for all methods except MPNN MVE across the evaluated datasets.The metric allows a linear adjustment before assessing agreement with observed absolute errors.
  • Calibration: Figure S5 aggregates calibration slopes across datasets and splits, where slopes far from 1 indicate poor calibration.The large number of slopes ≥10 for MPNN Ensemble and FFN Ensemble indicates consistent underestimation of uncertainty.

Wilcoxon

The paper uses a modified Wilcoxon Signed-Rank Test to compare UQ methods across datasets or splits and three evaluation metrics. Its signed z-score indicates whether the primary method tends to outperform the secondary method, accounting for whether higher or lower metric values are preferred.

  • Figure S6 reports pairwise WSRT z-scores for UQ methods across datasets and the Spearman coefficient, Miscalibration Area, and NLL metrics.Higher z-scores indicate that the primary estimator often ranked better than the secondary estimator on the specified metric.
  • The test compares scores for primary and secondary UQ metrics over n different datasets or splits.The primary scores are represented as x1, x2, · · ·, xn and the secondary scores as y1, y2, · · ·, yn.
  • WSRT z-scores are computed from ranked pairwise differences between the primary and secondary metric scores.The differences are sorted by magnitude, and the resulting ranks are used to calculate the statistic.
  • The sign of each difference is chosen according to whether larger or smaller metric values are desirable, while recording which method performed better.Positive values correspond to primary-method improvement when larger metric values are preferred; negative values apply when smaller values are preferred.
  • The standard signed-rank procedure is modified to produce a signed z-score distinguishing significant primary-method improvements from cases favoring the secondary method.The supplied method description states that the modification enables directional comparison between the two metrics.
Loading 2005.10036v1…