Source-linked AI summary
MoleculeNet: A Benchmark for Molecular Machine Learning
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, Vijay Pande
TL;DR
Molecular machine learning lacked a standard benchmark for comparing methods across datasets, so MoleculeNet assembled a broad benchmark of datasets, tasks, and evaluation workflows. Graph-based models outperformed other methods on most datasets, revealing an advantage for learnable featurizations.
Problem
Existing molecular databases were not machine-learning focused, leaving a need for a molecular machine-learning benchmark.
Method
MoleculeNet benchmarks molecular machine learning across 17 dataset collections, over 800 tasks, and 700,000 compounds.
Results
11/17 datasets favored graph-based models by comfortable margins, revealing a clear advantage of learnable featurizations.
Takeaways & Limitations
Learnable featurizations are broadly effective for molecular machine learning, though their advantage varies across datasets.
Takeaways & Limitations
Graph-based methods are not robust enough on complex tasks under data scarcity.
Abstract
from arXiv · showhide
Molecular machine learning has been maturing rapidly over the last few years. Improved methods and the presence of larger datasets have enabled machine learning algorithms to make increasingly accurate predictions about molecular properties. However, algorithmic progress has been limited due to the lack of a standard benchmark to compare the efficacy of proposed methods; most new algorithms are benchmarked on different datasets making it challenging to gauge the quality of proposed methods. This work introduces MoleculeNet, a large scale benchmark for molecular machine learning. MoleculeNet curates multiple public datasets, establishes metrics for evaluation, and offers high quality open-source implementations of multiple previously proposed molecular featurization and learning algorithms (released as part of the DeepChem open source library). MoleculeNet benchmarks demonstrate that learnable representations are powerful tools for molecular machine learning and broadly offer the best performance. However, this result comes with caveats. Learnable representations still struggle to deal with complex tasks under data scarcity and highly imbalanced classification. For quantum mechanical and biophysical datasets, the use of physics-aware featurizations can be more important than choice of particular learning algorithm.
Introduction
MoleculeNet addresses the lack of a standard evaluation platform for molecular machine learning by curating datasets, implementing featurizations and algorithms, and providing standardized data splitting and benchmark results.
- Motivation: MoleculeNet was created because molecular machine learning lacked a standard evaluation platform, making it difficult to determine whether proposed techniques improve performance.Algorithmic papers often benchmarked methods on disjoint dataset collections.
- Contributions: The benchmark curates dataset collections, implements known molecular featurizations and previously proposed learning algorithms, and integrates them into DeepChem.These resources are intended to facilitate comparison and development of new molecular machine learning methods.
- Benchmark resources: MoleculeNet contains data on the properties of over 700,000 compounds.The datasets are curated and integrated into the open source DeepChem package.
- Evaluation: MoleculeNet provides multiple data-splitting mechanisms and evaluates algorithms across multiple split choices because random splitting is often incorrect for chemical data.The benchmark reports results across various featurizations and splits on its dataset collections.
Related Work
Existing chemical databases have enabled computational chemistry but were not designed for machine-learning comparison: they lacked standardized metrics and prescribed data splits. MoleculeNet addresses this gap by benchmarking broad metrics, splits, and dataset subsets while incorporating and expanding earlier chemistry benchmarking efforts.
- Existing chemical databases: PubChem, PubChem BioAssay, ChEMBL, ChemSpider, crystallographic databases, and the Protein Data Bank provide large curated repositories of molecules, bioassays, bioactivities, structures, and compounds.These resources primarily support human-oriented browsing or repository access rather than standardized machine-learning evaluation.
- Limitations for machine learning: Previous databases were not machine-learning focused because they defined neither effectiveness metrics nor prescribed training/validation/test splits.Researchers therefore made independent choices about evaluation procedures when using subsets of these collections.
- Limitations for machine learning: Variable metric and subset choices made results from chemical machine-learning papers using the same data sources potentially incomparable.The passage specifically notes that two methods papers using PubChem data may be entirely incomparable.
- MoleculeNet’s contribution: MoleculeNet bridges this gap by providing benchmark results across a reasonable range of metrics, splits, and subsets of existing and other data collections.It also incorporates data from the Quantum Machine group and prior multitask-learning benchmarking efforts while significantly expanding them.
Methods
MoleculeNet is implemented through DeepChem as a configurable benchmarking system spanning public datasets, data splits, metrics, featurizations, and learning methods. Its collection covers diverse molecular-property categories and supports both SMILES-based and coordinate-based inputs.
- Benchmark implementation: DeepChem provides a benchmark interface in which users select datasets, splitting methods, featurizers, and models, including user-defined models.The benchmark can be run through deepchem.molnet.run benchmark(datasets, model, split, featurizer).
- Datasets: The collection contains 17 prepared and benchmarked datasets drawn from public databases, with coverage spanning quantum mechanics, physical chemistry, biophysics, and physiology.The datasets cover molecular properties at levels ranging from molecular-scale properties to macroscopic influences on the human body.
- Evaluation design: For each dataset, MoleculeNet specifies a recommended data-splitting pattern and evaluation metric, with results reported using those choices.Dataset details include tasks, compounds, features, recommended splits, and metrics.
- Input representations: Most datasets represent molecules with SMILES strings, while some also provide 3D coordinates as molecular features for methods that use them.Labels are binary for classification tasks and floating-point values for regression tasks.
- Featurization: MoleculeNet implements six molecular featurization methods, including a grid featurizer introduced in this work and learnable featurization processes.The benchmark explicitly provides implementations of multiple molecular featurization methods.
Results and Discussion
Across MoleculeNet results, learnable and physics-aware featurizations often improve performance, but data scarcity, overfitting, class imbalance, and computational cost limit model robustness. Graph-based models excel with sufficient data, whereas simpler methods can be more robust on small datasets.
- Classification benchmarks: Multitask training generally reduces overfitting and compensates for limited per-task data by combining tasks into a larger training pool.Singletask logistic regression shows the largest train–validation/test gaps, while multitask structure generally shows less overfit.
- Classification benchmarks: For smaller singletask datasets (<3000 samples), kernel SVM and ensemble trees are more robust under data scarcity, while graph models generally require more training data.Kernel SVM and ensemble trees require longer running time, whereas graph convolutional and weave models perform strongly on larger datasets.
- Classification benchmarks: Graph convolutional models perform best on PCBA, but results on MUV are much less stable, and graph-based models remain insufficiently robust at extreme class imbalance.PCBA has a positive rate of 1.40%, whereas MUV has 0.20%; both use AUC-PRC, which is more useful under high class imbalance.
Conclusion
MoleculeNet establishes a broad benchmark and open-source framework for molecular machine learning, showing that graph-based, learnable featurizations generally perform best while specialized features remain important for difficult tasks. Its results also identify limitations under data scarcity, class imbalance, and biophysical or physiological complexity, motivating broader benchmark coverage.
- Benchmark scope: MoleculeNet covers 17 dataset collections, over 800 tasks, and 700,000 compounds spanning four levels of molecular properties.The levels include quantum mechanical, physical chemistry, biophysical, and physiological properties.
- Benchmark results: Graph-based models achieved the best performance on 11/17 datasets, revealing a clear advantage for learnable featurizations.The comparison is summarized in Table 3 as conventional methods versus graph-based methods.
- Limitations: Graph-based methods are not robust enough on complex tasks under data scarcity, while kernel SVM outperforms learnable featurizations on imbalanced classification by positive recall.These caveats qualify the broad advantage of graph-based models.
- Task-specific modeling: Specialized featurizations are significant for PDBBind and quantum mechanics datasets, with distance-aware DTNN and MPNN outperforming simple graph convolutions on QM datasets.The conclusion emphasizes matching features to the information required by each task.
- Future directions: MoleculeNet provides a uniform comparison platform and open-source design intended to support new methods, broader molecular-property coverage, and community-contributed implementations.Future examples include 3D protein structure prediction and DNA topological modeling.
Appendix · Model Training and Hyperparameter Optimization
The appendix describes reproducible model training through DeepChem on Stanford GPU clusters and hyperparameter selection using Gaussian Process Optimization with pyGPGO.
- Model Training and Hyperparameter Optimization: Models were trained on Stanford’s GPU clusters via DeepChem.
- Model Training and Hyperparameter Optimization: Training was capped at 10 hours per model.
- Appendix: Benchmark users could reproduce the experiments locally by following DeepChem’s directions.
- Model Training and Hyperparameter Optimization: Hyperparameters were selected using Gaussian Process Optimization via pyGPGO.
- Model Training and Hyperparameter Optimization: The maximum number of hyperparameter-optimization iterations was set to 20.
- Appendix: Optimized hyperparameters for each model were listed, with additional details available separately.
Logistic Regression (Logreg) · Support Vector Classification (KernelSVM)
The supplied passage identifies the radial-basis-function kernel coefficient gamma as a method detail for Support Vector Classification (KernelSVM). No substantive passage is provided for Logistic Regression (Logreg).
- Support Vector Classification (KernelSVM): KernelSVM uses a radial-basis-function kernel coefficient named gamma.The passage lists gamma as a method detail for Support Vector Classification (KernelSVM).
- Support Vector Classification (KernelSVM): The passage specifically associates gamma with the radial basis function.This identifies the kernel context for the coefficient without providing a value or tuning result.
- Support Vector Classification (KernelSVM): Gamma is presented as a KernelSVM coefficient rather than as a reported evaluation metric.The passage contains no quantitative result, baseline, or candidate comparison.
- Support Vector Classification (KernelSVM): The supplied KernelSVM detail concerns the radial-basis-function kernel configuration.No additional implementation or performance information is given.
- Support Vector Classification (KernelSVM): The passage reports gamma as the only listed KernelSVM method detail.No gamma value or selection procedure is specified.
- Logistic Regression (Logreg): No passage supplied here reports a Logistic Regression (Logreg) method detail or result.The available evidence addresses only the KernelSVM radial-basis-function coefficient gamma.
Kernel Ridge Regression (KRR) · Random Forest (RF)
The supplied passage specifies a 500-tree configuration for the Random Forest (RF) method.
- Random Forest (RF): 500 trees were used in the Random Forest (RF).The passage lists the forest size as 500 trees.
Gradient Boosting (XGBoost)
The section identifies the number of boosted trees as a component of the XGBoost method.
- The XGBoost method includes the number of boosted trees.
Multitask/Singletask Networks … Influence Relevance Voting
The paper validates implementations across molecular neural-network architectures and datasets, reporting task-specific reimplementation results under specified data splits. It also documents model hyperparameters, benchmarking procedures, dataset access, featurization, and dataset construction details.
- Message Passing Neural Networks (MPNN): Final performances were run three times with fixed numerical seeds and deterministic data splitting, although same-seed benchmark runs may vary by about 1%.Mean and standard deviations are presented for the results.
- Performances: Reimplementation results are reported across classification, regression, and quantum-mechanical performance tables using task-specific AUC, RMSE, and MAE metrics.The tables cover PCBA, MUV, HIV, BACE, BBBP, Tox21, ToxCast, SIDER, ClinTox, PDBbind, ESOL, FreeSolv, Lipophilicity, QM7, QM7b, QM8, and QM9.
- ClinTox: ClinTox compares FDA-approved drugs with drugs that failed clinical trials for toxicity reasons.Approved drug names came from SWEETLEAD, while failed-trial drugs came from AACT records marked terminated, suspended, or withdrawn with toxicity-related explanations.
- Dataset and model access: DeepChem provides commands for loading MoleculeNet datasets and models, with further instructions and customized-dataset examples available in its documentation and notebooks.The listed models include scikit-learn-based models and XGBoost.
- Model validation: Validation compares MoleculeNet reimplementations with reported results, but differences can arise from unrestricted training time and different random splitting.The paper provides validation scripts and trained models in DeepChem.