Analyzing Learned Molecular Representations for Property Prediction

• Published results do not provide a clear answer: Wu et al. 2 report convolution-based models typically outperforming fingerprint-based models, while Mayr et al. 12 report the opposite.Source: Introduction · p. 2
• The proposed model combines convolutions with fixed descriptors and constructs encodings around bonds rather than atoms, avoiding unnecessary message-passing loops.This hybrid design combines task-specific learning with a fixed descriptor prior.Source: Introduction · p. 3
• Across over 850 experiments on 19 publicly available benchmarks and 16 proprietary datasets, Bayesian optimization provides a robust, automatic solution for hyperparameter selection, which crucially affects model performance.The proprietary datasets came from Amgen, Novartis, and BASF.Source: Introduction · pp. 3–4
• Under scaffold splits, model rankings are consistent across public and proprietary datasets, and scaffold splits approximate industry’s temporal splits better than random splits.Random splitting is a poor approximation of temporal evaluation, confirming Sheridan.13.Source: Introduction · p. 3
• Hybrid representations outperform convolution-based and fingerprint-based models, but fingerprint models can outperform learned representations on datasets with up to 1000 training molecules because of data sparsity.The hybrid representation generalizes better across dataset characteristics.Source: Introduction · pp. 3–4
• Our model achieves comparable or better performance on 11 out of 19 public datasets and 15 out of 16 proprietary datasets than all baseline models.Baseline rankings vary across datasets, whereas the proposed model remains consistently strong.Source: Introduction · p. 4

Background

• Conventional molecular property prediction applies support vector machines or random forests to expert-engineered descriptors and molecular fingerprints.Examples include Dragon descriptors and Morgan (ECFP) fingerprints.Source: Background · p. 4
• Graph convolutional neural networks learn expert feature representations directly from molecular graphs and can capture complex relationships given sufficient data.This model belongs to the graph-based category of molecular property-prediction architectures.Source: Background · p. 5
• Prior graph-neural-network work includes evaluations on private industry datasets, but direct comparison with Liu et al. 40 is precluded by dataset differences.The paper also situates its graph encoder among earlier graph-encoder research in social networks and chemistry applications.Source: Background · pp. 4–5
• The model builds on the Message Passing Neural Network framework presented by Gilmer et al. 4.Its message-passing paradigm updates representations of directed bonds rather than atoms.Source: Background · p. 5
• The model further combines computed molecule-level features with the molecular representation learned by message passing.This augments the learned graph representation with computed molecular information.Source: Background · p. 5

Message Passing Neural Networks

• An MPNN alternates T rounds of message passing, which builds a molecular representation from atom and bond features, with a readout that predicts one or multiple properties.The model is trained end-to-end by backpropagating loss gradients through both message passing and readout phases.Source: Message Passing Neural Networks · p. 6

Directed MPNN

• D-MPNN propagates hidden states along directed bonds rather than atoms, preventing immediate backtracking paths that can introduce noise into molecular representations.The directed message from 1 → 2 is propagated to nodes 3 and 4 in the next iteration, not back to node 1.Source: Directed MPNN · p. 7
• D-MPNN initializes directed-edge states from atom and bond features, updates them with learned functions and skip connections, then sums incoming bond features into atom representations for readout.The readout sums atom hidden states and applies a feed-forward neural network to generate property predictions.Source: Directed MPNN · pp. 7, 9

Initial Featurization

• Initial node and edge features are the corresponding atom and bond descriptors, computed with RDKit and represented primarily through one-hot encodings.Atomic mass is the stated exception, represented as a scaled real-valued feature.Source: Initial Featurization · p. 10, Table 1 · p. 10, Table 2 · p. 11

D-MPNN with Features

• D-MPNN with Features augments directed message passing with 200 rapidly computed global molecular features to mitigate overfitting and local representations on small datasets.The features are computed in silico using RDKit and are intended to provide information that the graph model may not learn from limited data or insufficient message-passing depth.Source: D-MPNN with Features · p. 11
• D-MPNN readout concatenates learned molecular features with computed global features before feed-forward prediction, a general approach applicable to any MPNN and descriptors.The prediction is defined as ŷ = f(cat(h, h_f)).Source: D-MPNN with Features · p. 12

Hyperparameter Optimization

• Bayesian optimization with Hyperopt tunes model depth, hidden size, feed-forward layers, and dropout probability to maximize performance.The motivation is that MPNN performance can depend strongly on hyperparameter settings.Source: Hyperparameter Optimization · p. 14

Ensembling

• The ensemble averages equally weighted predictions from several independently trained models with different random initializations, while direct prior-work comparisons use a single D-MPNN.Ensemble results are also reported to illustrate the model’s maximum possible performance.Source: Ensembling · p. 14

Implementation

• The model is implemented in PyTorch, with code for D-MPNN variants and RDKit feature CDFs released publicly alongside a web demonstration.The D-MPNN code is available through GitHub, Descriptastorus provides feature-CDF code, and the demonstration covers public datasets.Source: Implementation · p. 14

Data

• The evaluation uses 19 public datasets ranging from fewer than 200 to over 450,000 molecules, covering regression and classification targets in quantum mechanics, physical chemistry, biophysics, and physiology.The datasets come from Wu et al. 2 and Mayr et al..12; MUV contains only 0.2% positive molecules, making performance unstable.Source: Data · p. 15

Experimental Procedure

• Models use 20 iterations of Bayesian optimization over 10 randomly seeded 80:10:10 splits, selecting hyperparameters by validation performance and evaluating retrained models on test sets.Because of computational cost, only three splits are used for HIV, QM9, MUV, PCBA, and ChEMBL; evaluations include random and scaffold-based splits.Source: Experimental Procedure · p. 17

Results and Discussion

• The analysis varies data-splitting strategies and examines how split choice affects evaluation of generalization performance.The MoleculeNet comparison includes random, scaffold, and time splits, but original splits could not be reproduced for BACE, Toxcast, and QM7.Source: Results and Discussion · p. 19, Comparison to MoleculeNet · p. 21
• Statistical significance is evaluated with one-sided Wilcoxon signed-rank and one-sided Welch’s t-tests; Wilcoxon is stronger but requires paired molecule-level predictions unavailable for some models or split comparisons.Source: Results and Discussion · p. 20

Comparison to Baselines

• The best single optimized model is compared against models from prior work on each dataset, using public and proprietary evaluation settings.The broader analysis asks how performance compares with benchmarks and approaches experimental reproducibility.Source: Results and Discussion · p. 19, Comparison to Baselines · p. 20

Comparison to MoleculeNet

• D-MPNN uses one consistent architecture across datasets, whereas the best MoleculeNet architecture differs by dataset.This architectural consistency is an additional benefit of the D-MPNN comparison.Source: Comparison to MoleculeNet · p. 21
• D-MPNN may underperform on MUV because only 0.2% of samples are positive, making its extreme class imbalance unusually difficult.All other datasets contain at least 1% positives.Source: Comparison to MoleculeNet · p. 21
• D-MPNN outperforms the best graph-based MoleculeNet method on PDBbind and QM9, and the best MoleculeNet model on QM8 with or without 3D information.The current D-MPNN does not use 3D coordinate information, and extending it to use such information is left to future work.Source: Comparison to MoleculeNet · p. 21
• Across 10 datasets where MoleculeNet models use only 2D information, D-MPNN is significantly better than the best models on 5 datasets, not significantly different on 3, and significantly worse on 2.Source: Comparison to MoleculeNet · p. 21

Comparison to Mayr et al. 12

• Mayr et al. 12 performs similarly on most classification datasets but poorly on regression datasets, potentially because of binary inputs, absent early stopping, and resulting overfitting or instability.The binary-input explanation is supported by similarly poor performance from the Morgan fingerprint FFN baseline, while the early-stopping issue may cause training-data overfitting and numerical instability.Source: Comparison to Mayr et al. 12 · p. 23
• D-MPNN is significantly better than Mayr et al. 12 on 8 datasets, not significantly different on 10 datasets, and significantly worse on 1 dataset.The comparison reproduced Mayr et al.’s features using their scripts or equivalent packages, then ran their code and hyperparameter optimization directly.Source: Comparison to Mayr et al. 12 · p. 23

Out-of-the-Box Comparison of D-MPNN to Other Baselines

• Using all default settings—hidden size = 300, depth = 3, two feed-forward layers, dropout = 0, no hyperparameter optimization, and no additional features—D-MPNN provides an excellent starting point across a wide variety of datasets and targets.Source: Out-of-the-Box Comparison of D-MPNN to… · pp. 23, 25

Proprietary Datasets

• Across private industry datasets, the model’s strong public-dataset performance translated to real-world industrial datasets.The proprietary evaluations covered Amgen, BASF, and Novartis datasets.Source: Proprietary Datasets · p. 25, Amgen · p. 28, BASF · p. 30

Amgen

• On chronological Amgen splits, the D-MPNN outperformed baselines on 4 of 5 internal datasets, showing transfer from public scaffold splits to private industry data.The evaluated endpoints included rPPB, solubility, RLM, and hPXR datasets.Source: Amgen · pp. 25, 28

Novartis

• On the proprietary Novartis regression dataset, the D-MPNN outperformed the other baseline models.The evaluation used a chronological data split.Source: Novartis · p. 30, Figure 8 · p. 32

BASF

• On BASF internal regression datasets using a scaffold split, the D-MPNN outperformed all baseline models, and the authors report that it was robust enough to transfer to proprietary datasets.The BASF study used 10 highly related quantum mechanical datasets, each containing 13 properties calculated on the same 30,733 molecules, with solvents varying.Source: BASF · pp. 28, 30, Figure 7 · p. 31

Experimental Error

• The gap between model performance and experimental assay agreement indicates significant remaining space for future performance improvement.The comparison used R2 values on Amgen’s private regression datasets.Source: Experimental Error · p. 31
• Both the D-MPNN and Amgen’s internal expert-descriptor model remained far less accurate than the corresponding ground-truth assays.Experimental error was defined as agreement between multiple runs of the same assay and used as an upper-bound comparison.Source: Experimental Error · p. 31

Motivation for split choice

• Chronological splits best approximate future chemistry because they train on molecules before a cutoff date and test on molecules after it.This setup reflects molecules chemists are likely to investigate in the future, but chronological data are typically unavailable for public datasets.Source: Analysis of Split Type · p. 32

Comparison with chronological splits

• Scaffold-split performance is on average closer to chronological-split performance on proprietary Amgen and Novartis datasets and public PDBbind datasets.These comparisons are noisy because chronological evaluation provides only one split, whereas random and scaffold splitting include random variation.Source: Analysis of Split Type · p. 34

Evaluation policy

• The study reports all main results on scaffold splits to better reflect generalization to new chemistry, while additionally reporting chronological results wherever available.Chronological splits remain the ideal evaluation when available; sliding time-window splits reduce noise but substantially decrease dataset size and yield qualitatively similar conclusions in the Supporting Information.Source: Analysis of Split Type · p. 34

Overall findings

• Scaffold and chronological splits are more difficult than random splits, supporting scaffold splits over random splits for evaluation.The authors also find some evidence that chronological splits may be harder than scaffold splits, but the limited number of available chronological datasets warrants further investigation.Source: Analysis of Split Type · p. 34

Scaffold-overlap rationale

• Chronological train and test sets share fewer molecular scaffolds than random splits, while scaffold splits enforce zero scaffold overlap and should therefore be at least as difficult.Figure 10 compares scaffold overlap for random and chronological splits across four Amgen regression datasets.Source: Analysis of Split Type · pp. 32, 34, Figure 10 · p. 35

Message Type

• Directed-bond message passing outperforms atom- and undirected-bond alternatives on average, although most differences are not statistically significant.The authors conclude that further investigation of the message-passing paradigm is warranted.Source: Message Type · p. 35, Figure 15 · p. 40

RDKit Features

• RDKit features help all three PDBbind datasets, with the largest benefit on the extremely small PDBbind-C core dataset.This suggests features may compensate for limited training data and be especially useful in low-data regimes.Source: RDKit Features · p. 41, Figure 16 · p. 42
• RDKit features improve some datasets, including QM9 and ESOL, but worsen performance on others, including PCBA and HIV.The effect is task-dependent, potentially because features are relevant for some tasks but distracting for others.Source: RDKit Features · p. 39, Figure 16 · p. 42

Hyperparameter Optimization

• 37% better performance on QM9 follows Bayesian hyperparameter optimization, while most datasets improve by a moderate 2-5%.QM7, QM8, QM9, and PCBA show particularly dramatic improvements.Source: Hyperparameter Optimization · p. 41, Figure 17 · p. 43

Ablations

• Combining RDKit descriptors, hyperparameter optimization, and ensembling significantly improves performance on every dataset except MUV.Each optimization alone has limited benefits, but their combination yields substantial gains overall.Source: Ensembling · p. 44

Ensembling

• Ensembling five models provides a small 1-5% benefit on most datasets, with especially strong benefits on quantum mechanics datasets.The ensemble uses the best single-model hyperparameters and RDKit features when those features improve performance.Source: Ensembling · p. 41, Figure 19 · p. 45

Effect of Data Size

• D-MPNN may outperform the best Mayr et al. [12] model at larger data thresholds, but the results are not fully conclusive.The analysis uses ChEMBL targets filtered by increasing label-count thresholds, and all comparisons besides the first are statistically significant.Source: Figure 20 · p. 46, Effect of Data Size · p. 46

Conclusion

• D-MPNN consistently matched or outperformed each baseline individually and achieved comparable or better performance on 11 of 19 public datasets; on the remaining eight, no single baseline was consistently superior.Source: Conclusion and Future Work · p. 47
• On proprietary datasets, D-MPNN outperformed random forest, feed-forward neural network, and Mayr et al. 12 models on 15 of 16 datasets.These results support using learned molecular representations in industrial property prediction settings.Source: Conclusion and Future Work · p. 47

Future Work

• D-MPNN typically underperformed when competing models used 3D information, datasets were especially small, or classification classes were particularly imbalanced.Examples include MoleculeNet’s best QM and PDBbind models, PDBbind-C, and MUV.Source: Conclusion and Future Work · p. 47
• Future improvements include incorporating additional 3D information beyond the model’s currently very restricted and naive representation, and developing principled pretraining to transfer learning from large chemical datasets to smaller datasets.Such pretraining could improve performance in limited data settings.Source: Conclusion and Future Work · p. 47
• The analysis of split type and model generalizability motivates future work on uncertainty quantification and domain of applicability assessment.This direction follows from examining how split type affects estimated model generalizability.Source: Conclusion and Future Work · p. 47

Supporting Information Available

• The Supporting Information files are available free of charge.Source: Supporting Information Available · p. 48
• The Supporting Information links to the authors’ code and demonstrates their web-based user interface.Source: Supporting Information Available · p. 48
• It provides further model-performance comparisons using both scaffold-based and random data splits.Source: Supporting Information Available · p. 48
• Tables report all raw performance values, including the p-values underlying the paper’s charts.Source: Supporting Information Available · p. 48
• The materials analyze class balance across classification datasets.Source: Supporting Information Available · p. 49
• They also list the RDKit-calculated features used by the model.Source: Supporting Information Available · p. 49