PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning

Yao Zhang, Alpha A. Lee

arXiv:1902.00925v2cs.LGphysics.chem-phstat.ML

TL;DR

Molecular-property models need to identify unreliable predictions and use costly experiments efficiently, but common graph-neural-network evaluations emphasize mean error. This paper combines Bayesian uncertainty estimation with semi-supervised graph convolutional networks; it reports improved benchmark accuracy and uncertainty estimates, dataset-bias detection, and active learning.

  • Problem

    Mean test error alone does not tell users when predictions may fail, and costly experiments make informative training-set selection important.

    Source: §I · p. 2

  • Method

    Bayesian semi-supervised graph convolutional neural networks learn representations from unlabeled molecular structures and estimate uncertainty through Bayesian inference.

    Source: §I · p. 3, C. Bayesian deep learning · p. 7

  • Results

    The method significantly outperforms the baseline across benchmark problems in accuracy and uncertainty estimates, returns greater uncertainty for test molecules from a different chemical-space region, and supports active learning.

    Source: §IV · p. 18

  • Takeaways & Limitations

    Combining semi-supervised learning with Bayesian deep learning supports uncertainty estimation and active learning, including in the low-data setting.

    Source: §I · p. 3

    Bayesian inference is computationally expensive, and approximate methods trade off posterior quality against computational cost.

    Source: C. Bayesian deep learning · p. 7

Abstract

from arXiv · show

Predicting bioactivity and physical properties of small molecules is a central challenge in drug discovery. Deep learning is becoming the method of choice but studies to date focus on mean accuracy as the main metric. However, to replace costly and mission-critical experiments by models, a high mean accuracy is not enough: Outliers can derail a discovery campaign, thus models need reliably predict when it will fail, even when the training data is biased; experiments are expensive, thus models need to be data-efficient and suggest informative training sets using active learning. We show that uncertainty quantification and active learning can be achieved by Bayesian semi-supervised graph convolutional neural networks. The Bayesian approach estimates uncertainty in a statistically principled way through sampling from the posterior distribution. Semi-supervised learning disentangles representation learning and regression, keeping uncertainty estimates accurate in the low data limit and allowing the model to start active learning from a small initial pool of training data. Our study highlights the promise of Bayesian deep learning for chemistry.

I. INTRODUCTION

Molecular-property models must estimate when predictions may fail and use scarce experiments efficiently, not merely minimize mean error. The paper combines Bayesian uncertainty estimation with semi-supervised graph convolutional networks, whose message passing represents molecular structure.

I. INTRODUCTION

  • Graph neural networks take molecular graphs as input and learn the structure-to-descriptor map from data, outperforming classical methods with handcrafted descriptors.Source: §I · p. 2
  • Outliers among a small set of experimentally tested top-ranked predictions can derail discovery, while experiment costs motivate selecting informative training data.Source: §I · p. 2
  • Bayesian semi-supervised graph convolutional networks are proposed to estimate uncertainty in the low-data limit and drive active learning despite training-set bias.Source: §I · p. 3

A. Supervised graph convolutional neural network

  • The graph convolutional fingerprint model represents atoms and bonds as graph nodes and edges, using message passing to summarize each atom’s local environment while respecting atom-relabelling invariance.Source: A. Supervised graph convolutional… · p. 4

B. Semi-supervised graph convolutional neural network

Learning molecular descriptors directly from property-labelled data can be a disadvantage when active learning starts with little data. The semi-supervised approach learns representations from molecular structures, then uses supervised learning to predict properties.

B. Semi-supervised graph convolutional neural network

C. Bayesian deep learning

Bayesian inference represents uncertainty through a posterior distribution over model parameters and averages predictions over that posterior. Its uncertainty estimates include both parameter-related uncertainty and measurement noise, but evaluating the required integrals is computationally expensive.

C. Bayesian deep learning

  • Bayesian inference seeks a posterior distribution over model parameters given the data, and predicts for an unknown input by averaging over that posterior.Source: C. Bayesian deep learning · p. 6
  • Epistemic uncertainty is prediction variance with respect to the posterior, whereas aleatoric uncertainty is intrinsic measurement noise.Source: C. Bayesian deep learning · p. 6
  • When measurement variance is constant, the log posterior is equivalent up to a constant to mean-squared loss with the log prior as regularizer, making maximum-likelihood inference a special case of Bayesian inference.Source: C. Bayesian deep learning · p. 7
  • Evaluating the high-dimensional Bayesian integrals is computationally expensive, so the paper considers two approximate methods spanning a cost-accuracy spectrum.Source: C. Bayesian deep learning · p. 7

1. Dropout variational inference

Dropout variational inference approximates the parameter posterior with a distribution that is easier to sample. Repeated stochastic predictions provide the mean prediction and uncertainty components.

1. Dropout variational inference

  • Dropout is treated as approximate Bayesian inference, with the neural network forked at its last layer to predict both the target and its measurement variance.Source: §1 · p. 7
  • During training, units are randomly set to zero, yielding a Bernoulli-based distribution over model parameters that approximates the posterior.Source: §1 · p. 7
  • At prediction time, the model is run repeatedly with dropout, and the mean predictions and variances across runs provide the prediction and total uncertainty.The uncertainty combines epistemic and aleatoric components.Source: §1 · p. 7

2. Stein Variational Gradient Descent

Stein Variational Gradient Descent (SVGD) draws samples from the posterior through gradient descent rather than fitting a distribution to it. Its finite-particle approximation trades computational cost for interpolation between frequentist and full Bayesian inference.

2. Stein Variational Gradient Descent

  • SVGD draws posterior samples directly through gradient descent rather than fitting a distribution to the posterior.Equations (13)–(14) are described as an interacting-particle system that drives particles toward low-loss regions while forcing them apart to maximize entropy.Source: §2 · p. 8
  • With N = 1, the method recovers frequentist inference; as N → ∞, it samples exactly from the posterior, with this paper using N = 50.Computational cost and memory demands increase with N.Source: §2 · p. 8
  • Figure 1 compares wall-clock time on a Nvidia P100 GPU across graph convolution with dropout, semi-supervised graph convolution with dropout, and semi-supervised SVGD on the melting-point dataset.The comparison is plotted as a function of gradient-update steps.Source: §2 · pp. 8–9, Figure 1 · p. 9
  • SVGD updates N Stein particles per iteration, giving wall-clock time per iteration that scales as O(BMN).B is the number of training samples per iteration and M is the number of model parameters.Source: §2 · p. 9

3. Architectures and hyperparameters

The study uses common graph-convolutional neural-network settings from prior implementations rather than extensive hyperparameter optimization or architecture search. The selected architecture has two layers and 128 hidden units.

3. Architectures and hyperparameters

  • The study adopts common graph-convolutional neural-network parameters from the literature rather than extensive hyperparameter optimization or neural architecture search.Source: §3 · p. 9
  • The number of hidden neurons, nh, is selected by grid search over {32, 64, 128, 256}, using the best average five-fold cross-validation RMSE across datasets.The fingerprint dimension is kept at twice nh, and the hidden-layer count follows the original implementations.Source: §3 · p. 9
  • The selected architecture uses T = 3, N = 50, 128 hidden units, and two layers.Source: §3 · p. 10

D. Datasets

The benchmarks cover molecular physical-property regression and bioactivity prediction across six datasets. An XGBoost baseline on ECFP6 fingerprints ranks their difficulty from FreeSolv as easiest to p450 as hardest by R2.

D. Datasets

  • Physical-property benchmarks include 3,025 MeltingPoint measurements, 1,128 ESOL aqueous-solubility measurements, and 643 FreeSolv hydration-free-energy measurements.ESOL and FreeSolv are used in the MoleculeNet benchmark.Source: D. Datasets · p. 10
  • Bioactivity benchmarks include CatS for 595 molecules, Malaria for 13,417 molecules, and p450 for 8,817 molecules.The measurements are respectively log IC50, log EC50, and half-maximal effective concentration values.Source: D. Datasets · p. 10
  • By XGBoost ECFP6-fingerprint R2, dataset difficulty runs from easiest to hardest as FreeSolv, MeltingPoint, ESOL, CatS, Malaria, and p450.The baseline uses an 80/10/10 train/validation/test split and tunes the number of estimators on validation data.Source: D. Datasets · p. 10

III. RESULTS

Semi-supervised learning with Stein Variational Gradient Descent (SVGD) estimates prediction uncertainty across the tested datasets, with error decreasing as confidence rises. Combining epistemic and aleatoric uncertainty improves high-confidence predictions, while uncertainty-estimate quality also depends on posterior sampling.

A. Uncertainty quantification

  • Across every dataset, semi-supervised learning with SVGD significantly outperforms the baseline in uncertainty estimation, with test error decreasing as confidence increases.Figure 2 plots test-set error against confidence percentile; its shaded region is one standard deviation across five random train/test partitions.Source: Figure 2 · p. 11, A. Uncertainty quantification · pp. 11–12
  • For ESOL and FreeSolv, error falls steeply at low confidence, while for MeltingPoint, Malaria, and p450 it falls sharply near 100% confidence.The first pattern indicates a few outliers among mostly accurate predictions; the second indicates a few accurate predictions among many less accurate ones.Source: A. Uncertainty quantification · p. 12
  • At the same confidence percentile, SVGD consistently outperforms dropout, indicating that posterior-sampling quality strongly affects uncertainty estimates.Source: A. Uncertainty quantification · p. 12
  • Combining epistemic and aleatoric uncertainty gives the highest Spearman rank correlation between predicted variance and model error; this correlation broadly follows dataset difficulty.FreeSolv and ESOL have the highest rank correlations, while p450 has the lowest.Source: A. Uncertainty quantification · pp. 12–13, Table II · p. 13
  • Using both uncertainty sources produces confidence-error curves below or matching either source alone, with more accurate high-confidence predictions for ESOL and p450.Which source better estimates error varies by dataset: epistemic uncertainty does so for ESOL, while aleatoric uncertainty does so for p450 and CatS.Source: A. Uncertainty quantification · pp. 13–14, Figure 3 · p. 14

B. Overcoming dataset bias

Chemical datasets cluster around a few scaffolds, so random splits can make test molecules resemble training molecules and obscure poor generalization to unseen scaffolds. In a toy log P task, Bayesian uncertainty was higher for steroids absent from training, flagging predictions from the biased model.

B. Overcoming dataset bias

  • Random splits can yield optimistic test performance because molecules cluster around a few scaffolds, while unseen scaffolds remain difficult to predict.The practical question is whether uncertainty can flag likely errors for unseen molecules, rather than whether the data can be fully “unbiased.”Source: B. Overcoming dataset bias · pp. 14–15
  • The experiment trained on ChEMBL molecules with either beta-lactam or benzodiazepine scaffolds, then tested predictions on steroids absent from training.The model was reasonably accurate on the biased test set but performed poorly on steroids.Source: B. Overcoming dataset bias · p. 15
  • Estimated uncertainty was significantly greater for steroid predictions than for beta-lactam or benzodiazepine test-set predictions, signaling when the biased model was inaccurate.Figure 4 compares a model trained on beta-lactam and benzodiazepine scaffolds with predictions on steroids.Source: Figure 4 · p. 15, B. Overcoming dataset bias · p. 16

C. Low data active learning

The study evaluates active learning from a small initial training pool, adding compounds selected by epistemic uncertainty or at random. Semi-supervised learning outperformed full supervised learning in the low-data setting, but active learning depended on the initial set’s scaffold diversity.

C. Low data active learning

  • The active-learning cycle began with a random 25% of the training set, then added 2.5% per iteration by highest epistemic uncertainty or random sampling.The test set was held out at 20% of the full dataset; Figure 5 reports mean error curves over 20 runs.Source: C. Low data active learning · pp. 16–17
  • Semi-supervised learning significantly outperformed full supervised learning in the low-data limit, with active learning improving on random sampling only for the semi-supervised method.The comparison averages 20 active-learning runs; full supervised learning could not beat random sampling.Source: C. Low data active learning · p. 16, Figure 5 · p. 17
  • Active learning did not outperform random sampling when the initial Malaria training set was scaffold-biased and contained few scaffolds.With only one scaffold initially, the model had little knowledge of other scaffolds, making random sampling reasonable.Source: C. Low data active learning · p. 17, Figure 6 · p. 18
  • For the scaffold-bias test, the Malaria training set was scaffold-split to create a biased initial pool before comparing random and uncertainty-based additions.The experiment first set aside a random 20% test set, then scaffold-split the remaining training set.Source: C. Low data active learning · pp. 17–18

IV. DISCUSSION AND CONCLUSION

The method outperforms the baseline across benchmark problems in accuracy and uncertainty estimates, detects dataset bias, and supports active learning. Its performance is bounded by open questions about inference quality, uncertainty validation, and scaffold diversity.

IV. DISCUSSION AND CONCLUSION

  • The method outperforms the baseline on benchmark problems in model accuracy and uncertainty estimates, detects dataset bias, and supports active learning while minimizing training-set size.It returns large uncertainty when test compounds come from a different region of chemical space than the training set.Source: §IV · p. 18
  • The choice of Bayesian inference method affects uncertainty quality, while balancing computational cost, model accuracy, and uncertainty quality remains an open problem.Benchmarking approximate methods against mathematically exact posterior sampling with Markov Chain Monte Carlo is computationally expensive; existing benchmarks often emphasize RMSE or log-likelihood rather than confidence-error curves.Source: §IV · pp. 18–19
  • Whether the model accurately separates aleatoric from epistemic uncertainty remains unresolved without ground-truth uncertainty estimates from repeated measurements.Such measurements would report experimental variance; benchmark datasets with accurate experimental uncertainty estimates would help advance Bayesian methodology.Source: §IV · p. 19
  • Active learning performs well when the initial screen covers diverse compounds, but scaffold-biased starts require bioactivity information for unseen scaffolds.The authors suggest multitask learning across cognate assays that have explored those scaffolds as a possible avenue.Source: §IV · p. 19

Related papers

Loading 1902.00925v2…