MoleculeNet: A Benchmark for Molecular Machine Learning

Introduction

• Molecular machine learning faces scarce, costly data and must predict a broad range of properties from molecules with variable size, components, connectivity, and conformers.Source: Introduction · pp. 2–3
• Because conventional models expect fixed-length inputs, molecules require featurization into fixed-dimensional representations.Source: Introduction · p. 3
• MoleculeNet addresses inconsistent evaluation by providing a benchmark collection for comparing molecular machine learning methods.The paper identifies benchmarking on disjoint dataset collections as an obstacle to judging whether proposed methods improve performance.Source: Introduction · pp. 2–3
• MoleculeNet contains data on the properties of over 700,000 compounds, with datasets curated and integrated into the open-source DeepChem package.Users can access the benchmark data through DeepChem library calls.Source: Introduction · p. 4
• The benchmark provides high-quality implementations of established biochemical featurizations and previously proposed machine learning methods.These implementations are intended to facilitate comparison and development of new methods.Source: Introduction · p. 4
• Because random splitting is often unsuitable for chemical data, MoleculeNet supplies splitting mechanisms and evaluates algorithms across multiple split choices..Source: Introduction · p. 4

• These databases are not machine-learning focused and prescribe neither evaluation metrics nor train/validation/test splits, leaving choices variable and results potentially incomparable.Source: Related Work · p. 5
• MoleculeNet bridges this gap with benchmark results across a reasonable range of metrics, splits, and subsets of existing data collections.Source: Related Work · p. 5
• MoleculeNet incorporates benchmarking collections from the Quantum Machine group39 and prior multitask-learning work10, then significantly expands them.Source: Related Work · p. 5

Methods

• DeepChem’s molnet module benchmarks specified datasets, models, splits, and featurizers through a single call, while also supporting user-defined models.The benchmark script exposes different choices for data splitting, featurization, and learning.Source: Methods · p. 6

Datasets

• Most datasets represent molecules with SMILES, some include 3D coordinates, and labels are binary for classification or floating-point for regression.For each dataset, the benchmark recommends a metric and split suited to its properties.Source: Datasets · p. 7
• At the time of writing, MoleculeNet benchmarked 17 datasets from public databases, covering over 700,000 compounds across quantum mechanics, physical chemistry, biophysics, and physiology.The collection was intended to grow through further datasets and public contributions.Source: Datasets · pp. 6–7

QM7/QM7b

• Quantum benchmarks cover electronic properties from stable conformations, excited-state properties, and molecular geometry, with QM7 recommending stratified splitting.QM8 includes four excited-state properties calculated by three methods on 22,000 samples; QM9 integrates atomic coordinates as features.Source: QM7/QM7b · pp. 8–9

PCBA

• PCBA contributes 128 high-throughput bioassay tasks, while MUV provides 17 virtual-screening tasks and HIV labels combine confirmed and moderately active compounds as active.Scaffold splitting is recommended for HIV to support discovery of new inhibitor categories.Source: PCBA · p. 10

PDBbind

• PDBbind supplies 3D coordinates for ligands and target proteins to enable binding-geometry-aware featurization, with time splitting recommended to mimic field development.Its benchmark targets use the refined and core subsets.Source: PDBbind · pp. 10–11

BACE and BBBP

• BACE is a binary classification benchmark built from 1,522 compounds, for which scaffold splitting is recommended; BBBP concerns prediction of blood-brain barrier permeability.The BACE dataset combines compounds’ 2D structures with binary labels.Source: BACE · p. 11, BBBP · p. 11

BBBP

• BBBP covers binary permeability labels for over 2,000 compounds, and scaffold splitting is recommended for this well-defined target.Source: BBBP · p. 11

Tox21

• Tox21 provides qualitative toxicity measurements for 8,014 compounds across 12 targets, including nuclear receptors and stress-response pathways.The database was created by the Toxicology in the 21st Century initiative and used in the 2014 Tox21 Data Challenge.53Source: Tox21 · pp. 11–12

ToxCast

• ToxCast contributes qualitative results from over 600 in vitro toxicology experiments on a large compound library.54Source: ToxCast · p. 12

SIDER

• SIDER groups adverse drug reactions into 27 MedDRA system organ classes for 1,427 approved drugs.55,56,57Source: SIDER · p. 12

Dataset splitting

• MoleculeNet uses 80/10/10 training, validation, and test splits; scaffold splitting separates 2D structural frameworks and poses a greater challenge than random splitting.Training fits models, validation tunes hyperparameters, and test sets evaluate them.Source: Dataset splitting · pp. 12–13
• For QM7, stratified random sampling preserves the label range across splits; for PDBbind, time splitting trains on older data and tests on newer data.MoleculeNet contributes these split implementations to DeepChem for reuse.Source: Dataset splitting · pp. 13–14

Metrics

• With high class imbalance, PRCs and ROC curves can differ substantially because PRCs emphasize low recall, where false positives weigh more heavily.The recommended metric therefore depends on the experiment’s positive rate; the paper notes virtual screening often has extremely low positive rates.Source: Metrics · pp. 15–16
• MoleculeNet recommends PRC-AUC for datasets with positive rates below 2% and ROC-AUC otherwise.Regression datasets use MAE and RMSE; classification datasets use ROC-AUC and PRC-AUC, averaging metric values across tasks when datasets contain multiple tasks.Source: Metrics · pp. 14, 16

Coulomb Matrix

• The Coulomb Matrix encodes interatomic repulsion off-diagonal and atomic self-energy on-diagonal; it is translation- and rotation-invariant, but not atom-permutation-invariant.Random atom-index sorting and binary expansion can be applied during training to achieve atom-index invariance, as reported by Montavon et al. [18].Source: Coulomb Matrix · p. 18

Grid Featurizer

• The new grid featurizer represents protein–ligand binding-pocket interactions alongside ligand and protein features for PDBbind.It includes salt bridges, hydrogen bonds, ligand and protein circular fingerprints, and protein–ligand SPLIF fingerprints.Source: Grid Featurizer · pp. 18–19

Symmetry Function

• Symmetry functions encode local atomic environments with radial distance and angular terms, preserving rotational and permutation symmetry but initially distinguishing atom types poorly.MoleculeNet uses a modified version that separates radial and angular terms by atom type in pairs or triplets [71].Source: Symmetry Function · p. 19

Graph Convolutions; Weave

• Graph convolutions pair atom-level chemical features with molecular neighbor lists, while Weave represents connectivity with atom-pair features including bonds, graph distance, and ring information.Weave supports graph models using both atom nodes and bond edges.Source: Graph Convolutions · p. 19, Weave · p. 20

Models - Conventional Models

• MoleculeNet benchmarks conventional and graph-based methods, including logistic regression, support vector classification, kernel ridge regression, random forests, gradient boosting, multitask networks, and graph models.The methods are implemented in the open-source DeepChem package [30].Source: Models - Conventional Models · p. 20

Logistic Regression; Support Vector Classification; Kernel Ridge Regression; Influence Relevance Voting

• Logistic regression predicts from weighted features; RBF-kernel SVMs use regularization, kernel ridge regression learns nonlinear mappings, and IRV combines labels from the top K similar compounds.IRV is a refined K-nearest-neighbor classifier based on the hypothesis that compounds with similar substructures have similar functionality.Source: Logistic Regression · p. 20, Support Vector Classification · p. 21, Kernel Ridge Regression · p. 21

Random Forests; Gradient Boosting

• Random-forest predictions average outputs from trees trained on subsamples; gradient boosting instead sequentially incorporates relatively simple trees, generating each new tree greedily.The passage describes random-forest benchmarking as computationally intensive and identifies DeepChem’s XGBoost implementation for gradient boosting.79Source: Random Forests · p. 21, Gradient Boosting · p. 21

Multitask/Singletask Network; Bypass Multitask Networks

• Multitask networks share processed features across task-specific output heads, whereas bypass networks add independent task-specific input-to-output layers to address unrelated task variations [74].The shared representation is intended to enable information sharing across tasks [10].Source: Multitask/Singletask Network · p. 22, Bypass Multitask Networks · p. 22

Influence Relevance Voting

• Influence Relevance Voting ranks compounds by fingerprint Jaccard-Tanimoto similarity and predicts from the labels of the top K compounds using learned weights.A one-hidden-layer neural network produces the weights from the top compounds’ similarities and rankings.Source: Influence Relevance Voting · p. 23

Models - Graph Based Models

• MoleculeNet implements graph-based models that learn molecular features by applying adaptive functions to nodes and edges across different molecular graph variants.Figure 6 illustrates graph convolution, directed acyclic graph, Weave, message-passing, distance-based DTNN, and ANI-1 architectures.Source: Models - Graph Based Models · p. 23, Figure 6 · p. 24

Graph Convolutional models

• Graph convolutional models learn task-adapted atom representations by applying shared functions to atoms and bonded neighbors, then summing atom features for prediction.The implementation stacks graph-convolution, batch-normalization, and graph-pooling modules before a dense layer.Source: Graph Convolutional models · p. 25

Weave models

• Weave models update atom features using all other atoms and their pairs, enabling distant information transfer at the cost of increased complexity.Their implementation updates atomic and pair features through fully connected layers, then gathers atomic features into molecular features.Source: Weave models · pp. 25–26

Directed Acyclic Graph models

• Directed acyclic graph models orient bonds toward each possible central atom, propagate features toward it, and sum the resulting graph features for prediction.They evaluate one graph per atom, increasing the required calculations.Source: Directed Acyclic Graph models · p. 26

Deep Tensor Neural Networks

• Deep Tensor Neural Networks embed atom numbers and update atomic features using other atoms’ features and physical distances, rather than graph distances.Using 3D coordinates for those distances limits DTNNs to quantum-mechanical or possibly biophysical applications.Source: Deep Tensor Neural Networks · pp. 26–27

ANI-1

• ANI-1 builds atom-specific symmetry-function features from atom types and interactions, then uses atom-type-specific neural network potentials to predict properties.The benchmark uses molecules in their most stable configuration, unlike the original training setup with multiple poses; comparable accuracy is therefore not expected, and larger-training-set comparisons are left for future work.Source: ANI-1 · p. 27

Message Passing Neural Networks

• Message Passing Neural Networks stack message-passing phases to extract graph information, then use a readout phase to map the graph to properties.MoleculeNet reimplements an edge-network message function with set2set readout, using neighbor-atom features as inputs.Source: Message Passing Neural Networks · p. 28
• Updated messages are merged with gated recurrent units, while readout treats atom feature vectors as a set and applies multi-step LSTM attention, exporting the final state as outputs.Source: Message Passing Neural Networks · p. 28

Results and Discussion

• Model choices and featurizations varied with dataset size, features, and task type; each dataset–model combination received brief Gaussian-process hyperparameter optimization and three independent runs.Non-graph models used ECFP by default, with Coulomb Matrix and grid featurizations also applied for certain datasets.Source: Results and Discussion · p. 28

Physiology and Biophysics Tasks

• Most models overfit, but multitask models generally showed smaller train–validation/test gaps, consistent with a regularizing effect and their ability to pool data across low-volume tasks.Graph convolutional and weave models also showed strong validation/test results with less overfitting on larger datasets.Source: Physiology and Biophysics Tasks · p. 31
• For singletask datasets under 3,000 samples, kernel SVMs and ensemble trees were more robust under data scarcity, whereas graph models generally require more training data.The classical methods generally took longer to run.Source: Physiology and Biophysics Tasks · p. 32
• Graph convolution performed best on PCBA, but MUV results were less stable at a 0.20% positive rate, and graph models remained weak at controlling false positives under extreme imbalance.PCBA’s positive rate was 1.40%.Source: Physiology and Biophysics Tasks · p. 32
• On Tox21, every model improved with more training data, while graph models stayed on top and matched multitask networks using 30% rather than 90% of the training samples.Figure 10 reports mean out-of-sample performance over five runs, with standard deviations shown as error bars.Source: Physiology and Biophysics Tasks · pp. 32, 34, Figure 10 · p. 33

Biophysics Task - PDBbind

• As dataset size increased, validation/test performance significantly improved; on refined and full subsets, grid featurization modestly outperformed ECFP in single-task networks and random forests, while core showed no clear featurization difference.Source: Biophysics Task - PDBbind · p. 34
• PDBBind models showed heavy overfitting, and the full-set models were not significantly better than models trained with less data, possibly because the additional data were less clean.The paper notes that additional clean data may be needed for more accurate protein–ligand binding models.Source: Biophysics Task - PDBbind · p. 34

Physical Chemistry Tasks

• On physical chemistry tasks, graph-based methods significantly outperformed vanilla singletask networks, while differences among graph methods were minor and task-specific.The best models reached the stated ab-initio prediction accuracy levels: +/- 0.5 for ESOL and +/- 1.5 kcal/mol for FreeSolv.Source: Physical Chemistry Tasks · pp. 34, 36
• A weave model reached state-of-the-art FreeSolv free-energy performance with 200 samples and matched single-task accuracy with one-third as many training samples.Convolutional and weave models achieved better performance with 50% and 30% of the data, respectively.Source: Physical Chemistry Tasks · p. 37

Quantum Mechanics Tasks

• Distance-aware methods improved performance and reduced overfitting; graph-based methods usually beat multitask networks and KRR with Coulomb Matrix features.KRR and multitask networks with Coulomb Matrix features substantially beat their baselines on QM7 and QM9, while ANI-1, DTNN, and MPNN had lower error than graph convolutional models.Source: Quantum Mechanics Tasks · p. 37
• DTNN outperformed KRR with Coulomb Matrix features on 12/14 QM7b tasks, although averaging errors across different magnitudes reversed the mean-error comparison.ANI-1, DTNN, and MPNN collectively covered the best-performing models on 28/39 tasks in the category.Source: Quantum Mechanics Tasks · pp. 37, 40
• ANI-1, DTNN, and MPNN covered the best-performing models on 28/39 quantum-mechanics tasks, reflecting the strength of learnable featurization.Figure 14 summarizes benchmark performance across QM7, QM7b, QM8, and QM9.Source: Figure 14 · p. 39, Quantum Mechanics Tasks · p. 40
• On QM7 atomization energy, adding spatial-position information created a large performance gap, and ANI-1 achieved the lowest overall mean absolute error.DTNN and multitask networks with Coulomb Matrix features reached similar accuracy to previously reported work, though the MoleculeNet implementations still lagged those numbers.Source: Figure 15 · p. 40, Quantum Mechanics Tasks · p. 40
• For QM datasets, featurization choice is critical because Coulomb Matrix and ANI-1, DTNN, and MPNN representations encode charges and physical distances, unlike ECFP graph substructures.The passages identify charges and physical distances as required inputs for solving the Schrödinger equation.Source: Quantum Mechanics Tasks · pp. 40–41

Conclusion

• Graph-based models outperformed conventional methods on 11/17 datasets, demonstrating a broad advantage for learnable featurizations.The comparison summarizes test-subset performance.Source: Table 3 · p. 41, Conclusion · p. 42
• MoleculeNet covers 17 dataset collections, over 800 tasks, and 700,000 compounds spanning quantum, physical-chemistry, biophysical, and physiological properties.The tasks are grouped into four levels.Source: Conclusion · p. 41
• MoleculeNet contributes data-loading, featurization, splitting, and learning methods to DeepChem, with interchangeable components applied across its datasets.The authors hope its uniform comparison platform will facilitate new chemistry and machine-learning methods.Source: Conclusion · pp. 41–42
• Graph-based methods struggle on complex tasks with scarce data, while kernel SVMs outperform learnable featurizations in positive-class recall on highly imbalanced datasets.DTNN outperforms KRR(CM) on 14/16 QM7b tasks, although differing label magnitudes skew mean-MAE.Source: Conclusion · pp. 41–42
• For PDBBind and quantum-mechanics datasets, pertinent specialized features matter; DTNN and MPNN distance information improves QM performance over simple graph convolutions.The results emphasize specialized features for different tasks.Source: Conclusion · p. 42
• On FreeSolv, data-driven graph models can outperform ab-initio physical calculations with moderate data, suggesting increasing importance for data-driven physical chemistry.The comparison concerns prediction of solvation energies.Source: Conclusion · p. 42
• Biophysical and physiological results remain weaker than those for other datasets, suggesting that better featurizations or more data may be needed for broad utility.The authors hope to extend MoleculeNet to cover a broader range of molecular properties.Source: Conclusion · p. 42
• The authors envision broader coverage, including protein-structure prediction and DNA topological modeling, and invite contributions of novel algorithm implementations.They hope the open-source benchmark will grow into a comprehensive molecular machine-learning resource.Source: Conclusion · pp. 42–43

Model Training and Hyperparameter Optimization

• Models were trained on Stanford’s GPU clusters via DeepChem, and none was allowed to train for more than 10 hours.The paper points to Table 4 for the training time profile.Source: Model Training and Hyperparameter… · p. 43
• Training time profiles are reported in Table 4.Source: Model Training and Hyperparameter… · p. 43
• Users can reproduce the benchmarks locally by following directions from DeepChem.Source: Model Training and Hyperparameter… · p. 43
• Hyperparameters were determined using Gaussian Process Optimization via pyGPGO.Source: Model Training and Hyperparameter… · p. 43
• Gaussian Process Optimization used a maximum of 20 iterations.Source: Model Training and Hyperparameter… · p. 43
• Optimized hyperparameters for each model are listed, with detailed hyperparameters available as well.Source: Model Training and Hyperparameter… · p. 44

Support Vector Classification (KernelSVM)

• The Support Vector Classification passage lists gamma as the radial basis function kernel coefficient.Source: Support Vector Classification… · p. 44

Random Forest (RF)

• The Random Forest configuration used 500 trees.Source: Random Forest (RF) · p. 44

• The number of boosted trees is identified as an XGBoost parameter.Source: Gradient Boosting (XGBoost) · p. 44

Multitask/Singletask Networks; Bypass Networks; Influence Relevance Voting (IRV); Graph Convolutional models (GC); Weave models; Deep Tensor Neural Networks (DTNN); Directed Acyclic Graph models (DAG); Message Passing Neural Networks (MPNN)

• Model settings include weight and bias initialization, layer sizes, neighbor counts, feature lengths, distance bins, graph propagation limits, and message-passing and readout steps.The listed settings span multitask/singletask, bypass, IRV, GC, Weave, DTNN, DAG, and MPNN models.Source: Multitask/Singletask Networks · p. 45, Bypass Networks · p. 45, Influence Relevance Voting (IRV) · p. 45

Grid Featurizer; ClinTox; Dataset and model access

• The grid featurizer produces a 2052-length ligand–protein vector, while ClinTox distinguishes FDA-approved drugs from drugs that failed trials for toxicity reasons.The section also points readers to DeepChem commands for loading MoleculeNet datasets and models.Source: Grid Featurizer · p. 53, ClinTox · p. 54, Table 13 · p. 55

Model validation; Performances

• The authors report that reimplementation performance may differ from benchmark tables because they used more epochs and different random splits.The supplied performance tables cover classification, regression, and quantum-chemistry tasks; their captions do not provide cell values.Source: Table 5 · p. 49, Table 6 · p. 50, Table 8 · p. 51

Graph Convolutional models; Directed Acyclic Graph models

• On ESOL logS, GC reimplementation RMSEs were 0.31 for the test subset and 0.39 for the valid subset; DAG results were 0.58 and 0.68, respectively.Both reimplementations used an 80/10/10 random split, while the original studies reported cross-validation results.Source: Graph Convolutional models · p. 56, Directed Acyclic Graph models · p. 56

Weave models

• For Tox21, the Weave reimplementation scored 0.843 for the test subset and 0.857 for the valid subset, compared with original results of 0.846–0.867 across model structures.The original result was the median score from 5-fold cross-validation; the reimplementation used an 80/10/10 random split.Source: Weave models · pp. 56–57

Deep Tensor Neural Network

• For QM9 atomization energy, DTNN’s original result was 0.93 ± 0.02, while reimplementation results were 1.26 for the test subset and 1.15 for the valid subset.The original result used 100,000 training samples; the reimplementation used 106,400 training samples.Source: Deep Tensor Neural Network · p. 57

Message Passing Neural Network

• For the QM9 HOMO-LUMO gap task, the MPNN original result was 0.0544, versus 0.101 for the test subset and 0.0997 for the valid subset.The authors could not fully repeat the original result because its hyperparameters were unspecified; they compared one representative QM9 task.Source: Message Passing Neural Network · p. 57

Influence Relevance Voting

• On HIV, the IRV reimplementation achieved 0.852 for the test subset and 0.840 for the valid subset.The original evaluation used 10-fold cross-validation, whereas this reimplementation used an 80/10/10 random split.Source: Influence Relevance Voting · p. 58