Source-linked AI summary
Analyzing Learned Molecular Representations for Property Prediction
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, Tommi Jaakkola, Klavs Jensen, Regina Barzilay
TL;DR
The paper asks whether learned molecular representations outperform fixed fingerprints or descriptors when generalizing to new chemical space and whether public benchmarks reflect industrial settings. It benchmarks models across public and proprietary datasets and introduces a graph convolutional model with learned and fixed features. The proposed model matches or outperforms baselines broadly, while experimental reproducibility remains below the desired level.
Problem
Existing evidence does not clearly determine which molecular representation generalizes better to new chemical space, and industrial comparisons with employed models are limited.
Method
The paper benchmarks neural architectures across public and proprietary datasets and evaluates the proposed D-MPNN against fixed-representation and learned-representation baselines.
Results
The proposed model achieves comparable or better performance on 11 of 19 public datasets and performs similarly on 15 of 16 proprietary datasets.
Takeaways & Limitations
Learned molecular representations are applicable to drug-discovery property prediction, with the proposed model offering improvements over baseline models across datasets.
Takeaways & Limitations
The authors report that learned molecular representations have not yet reached the accuracy and reproducibility of experimental screening results.
Abstract
from arXiv · showhide
Advancements in neural machinery have led to a wide range of algorithmic solutions for molecular property prediction. Two classes of models in particular have yielded promising results: neural networks applied to computed molecular fingerprints or expert-crafted descriptors, and graph convolutional neural networks that construct a learned molecular representation by operating on the graph structure of the molecule. However, recent literature has yet to clearly determine which of these two methods is superior when generalizing to new chemical space. Furthermore, prior research has rarely examined these new models in industry research settings in comparison to existing employed models. In this paper, we benchmark models extensively on 19 public and 16 proprietary industrial datasets spanning a wide variety of chemical endpoints. In addition, we introduce a graph convolutional model that consistently matches or outperforms models using fixed molecular descriptors as well as previous graph neural architectures on both public and proprietary datasets. Our empirical findings indicate that while approaches based on these representations have yet to reach the level of experimental reproducibility, our proposed model nevertheless offers significant improvements over models currently used in industrial workflows.
Introduction
The paper addresses uncertainty about learned molecular representations by evaluating neural architectures under generalization-focused splits and introducing a hybrid graph model. Across public and proprietary datasets, the model is consistently competitive, while evaluation design and data scale materially affect conclusions.
- Motivation: The study targets unclear comparisons between learned molecular representations and fixed fingerprints or descriptors, especially for generalization to new chemical space.Prior findings differed partly because evaluation setups and dataset construction varied.
- Motivation: Scaffold overlap between training and test data must be explicitly controlled because neural models can memorize training molecular scaffolds and fail on new ones.The paper therefore treats scaffold-aware evaluation as central to meaningful property-prediction assessment.
- Approach: The paper introduces a hybrid model combining learned convolutions with fixed descriptors, using bond-centered convolutions to construct molecular encodings.The hybrid representation combines task-specific learning with a fixed descriptor prior while avoiding unnecessary message-passing loops.
- Evaluation: The benchmark comprises over 850 experiments across 19 public benchmarks and 16 proprietary datasets from Amgen, Novartis, and BASF.The evaluation tests whether public-dataset rankings represent rankings on industrial datasets.
- Findings: Under scaffold splits, model rankings are consistent across public and proprietary datasets, and scaffold splits approximate industry-style temporal splits better than random splits.Random splits are described as poor approximations of temporal evaluation.
- Findings: The hybrid representation generalizes better than convolution-based or fingerprint-based models, but fingerprint models can outperform learned representations on datasets with up to 1000 training molecules.Data sparsity negatively impacts learned representations on small datasets.
Background
Molecular property prediction spans fixed expert-crafted representations and models that learn directly from molecular structure. This work belongs to graph convolutional approaches and extends message passing with bond updates and computed molecular features.
- Fixed representations: Traditional molecular property prediction commonly applies support vector machines or random forests to expert-engineered descriptors and molecular fingerprints.Examples include Dragon descriptors and Morgan (ECFP) fingerprints.
- Related directions: Prior work has also explored architecture improvements, SMILES-based models, explicit 3D coordinates, and evaluations on private industry datasets.Direct comparison with one private-dataset method is unavailable because the datasets differ.
- Learned representations: Graph convolutional neural networks learn expert feature representations directly from molecular data and can capture complex relationships given sufficient data.They operate on the underlying molecular graph rather than only fixed descriptors or fingerprints.
- Proposed architecture: The proposed model builds on the Message Passing Neural Network framework by updating representations of directed bonds rather than atoms.This bond-centered message-passing design is the model’s distinguishing architectural choice.
- Proposed architecture: The model further combines computed molecule-level features with the molecular representation learned by message passing.This connects learned graph encodings with fixed computed features.
Methods
The paper describes message passing neural networks for molecular property prediction and develops a directed variant that passes messages between directed bonds. It also augments this representation with global molecular features to address locality and limited-data constraints.
- Message Passing Neural Networks: MPNNs operate on molecular graphs in message-passing and readout phases, building a molecular representation before predicting one or more properties.Training is end-to-end through the prediction, readout, and message-passing phases.
- Directed MPNN: D-MPNN uses directed edges to prevent backtracking paths such as v1v2 ··· vn where vi = vi+2, thereby avoiding unnecessary message-passing loops.The design is motivated by the possibility that such excursions introduce noise into the graph representation.
- Directed MPNN: D-MPNN updates hidden states associated with directed bonds rather than atom-centered hidden states, while preserving a molecular readout phase.Message direction matters, and each directed bond excludes dependence on its reverse message.
- Directed MPNN: Initial directed-bond hidden states combine atom and bond features through a learned matrix and ReLU, while updates include a skip connection to the original edge features.The final atom representation sums incoming directed-bond states before the readout generates property predictions.
- Extensions and limitations: The base MPNN representation is fundamentally local, so it may struggle with properties depending heavily on global molecular information.Limited datasets also constrain feature learning and increase susceptibility to overfitting, while T < diam(G) limits communication across distant atoms.
- Extensions and limitations: A D-MPNN variant incorporates 200 rapidly computed global molecular features from RDKit, using feature normalization to make their distributions comparable.The approach is presented as a general way to combine external information with MPNNs and computed descriptors.
Experiments
The experiments evaluate molecular property models across diverse public datasets using multiple data splits, cross-validation, hyperparameter optimization, and descriptor- or fingerprint-based baselines.
- Datasets: 19 public datasets span regression and classification targets across quantum mechanics, physical chemistry, biophysics, and physiology.
- Datasets: Dataset sizes range from fewer than 200 molecules to more than 450,000 molecules.
- Optimization: Hyperparameters are selected using 10 randomly seeded 80:10:10 splits and Bayesian optimization, followed by retraining with the selected settings.
- Data Splits: Models are evaluated on random, scaffold-based, and original prior-work splits, with chronological splits preferred when available.
- Data Splits: Scaffold splits are treated as more challenging and realistic than random splits and as a proxy for chronological generalization when temporal information is unavailable.
- Baselines: Baselines include MoleculeNet models, Mayr et al.’s expert-fingerprint network, random forests, and feed-forward networks using Morgan fingerprints or RDKit descriptors.
Results and Discussion
The study compares D-MPNN with prior models under several evaluation and significance-testing procedures, finding broad advantages on 2D public benchmarks while identifying important data and representation limits.
- Research Questions: The study asks how D-MPNN compares with public benchmarks, how split choice affects generalization evaluation, and which model elements maximize performance.
- Evaluation: Statistical significance is defined as p-value less than 0.05 and assessed with one-sided Wilcoxon signed-rank or Welch’s t-tests.
- Public Benchmarks: On 10 datasets where comparison models use only 2D information, D-MPNN is significantly better on 5, indistinguishable on 3, and significantly worse on 2.
- Limitations: D-MPNN may underperform on extremely imbalanced MUV and on tasks with auxiliary 3D information because the current model does not use 3D coordinates.
- Public Benchmarks: Despite these limits, D-MPNN outperforms the best graph-based MoleculeNet method on PDBbind and QM9 and the best MoleculeNet model on QM8 with or without 3D information.
Comparison to Mayr et al. 12
The paper compares D-MPNN with Mayr et al.’s descriptor-based network and with untuned fingerprint and descriptor baselines, reporting stronger overall performance and useful out-of-the-box behavior.
- Mayr Comparison: Mayr et al.’s baseline performs poorly on regression datasets despite extensive tuning, while achieving similar performance on most classification datasets.
- Mayr Comparison: D-MPNN is significantly better than Mayr et al.’s model on 8 datasets, not significantly different on 10, and significantly worse on 1.
- Mayr Comparison: D-MPNN generally outperforms Mayr et al.’s model, especially on regression datasets.
- Out-of-the-Box Comparison: The unoptimized D-MPNN uses default settings without hyperparameter optimization or additional features and is compared with fingerprint- and descriptor-based baselines.
- Out-of-the-Box Comparison: Without optimization, D-MPNN provides an excellent starting point across varied datasets and targets, while optimization can improve it further.
Proprietary Datasets
The D-MPNN is evaluated on proprietary datasets from Amgen, BASF, and Novartis using chronological or scaffold splits, and its public-dataset performance generally transfers to these industrial settings.
- Amgen: On four Amgen internal datasets evaluated with chronological splits, D-MPNN outperforms baselines on 4 out of 5 datasets.
- Amgen: The Amgen results indicate that strong performance on public scaffold splits can translate to chronological splits of private industry datasets.
- BASF: The BASF evaluation covers 10 related quantum-mechanical datasets based on 30,733 molecules, using scaffold-based splits because chronological data were unavailable.
- BASF: On the BASF datasets, D-MPNN is robust enough to transfer to proprietary data and outperforms the compared baselines.
- Novartis: On the proprietary Novartis dataset, D-MPNN outperforms the other baselines.
- Experimental Reproducibility: D-MPNN and Amgen’s internal descriptor model remain substantially less accurate than the corresponding ground-truth assays, leaving room for future improvement.
Analysis of Split Type
The paper evaluates scaffold splits as a proxy for chronological evaluation of generalization to new chemistry. Scaffold and chronological splits are generally harder than random splits, although chronological comparisons are noisy and remain limited.
- Evaluation rationale: Chronological splits approximate prediction on molecules chemists may investigate in the future, while scaffold splits are examined as a proxy when chronological data are unavailable.Chronological training uses molecules before a date and testing uses molecules after it; scaffold splits enforce zero scaffold overlap.
- Evaluation rationale: Chronological train–test sets share fewer molecular scaffolds than randomly split sets, motivating scaffold-based evaluation.The scaffold split enforces zero molecular scaffold overlap and is intended to be at least as difficult as chronological splitting.
- Results: Scaffold performance is generally closer to chronological performance on Amgen, Novartis, and PDBbind datasets.The authors note that chronological splitting is noisy because only one chronological split is available, whereas random and scaffold splits include a random component.
- Results: Scaffold splits generally produce more difficult and potentially more useful evaluations than random splits, so the paper reports results primarily on scaffold splits.Chronological splits remain the ideal evaluation when available.
- Interpretation: The results confirm that scaffold and chronological splits are harder than random splits, while evidence that chronological splits may be harder than scaffold splits remains limited.The authors call for investigation across a larger range of datasets.
Ablations
The ablations examine message passing, molecule-level features, optimization, ensembling, and data size. Directed-bond messages generally perform best, optimization and combined enhancements improve performance broadly, while low-data settings remain challenging.
- Message passing: Directed-bond message passing outperforms atom- and undirected-bond alternatives on average, but most differences are not statistically significant.The authors state that further investigation is warranted.
- Molecule-level features: Feature effects are dataset-dependent: QM9 and ESOL improve, whereas PCBA and HIV show worse performance after feature addition.The authors hypothesize that task relevance determines whether features help or distract the model.
- Molecule-level features: RDKit molecule-level features help across PDBbind datasets, with the largest benefit on the extremely small PDBbind-C dataset.The authors suggest these features may compensate for limited training data.
- Optimization: 37% better performance on QM9 follows Bayesian hyperparameter optimization, compared with the typical 2–5% improvement on most datasets.QM7, QM8, QM9, and PCBA are described as especially sensitive to optimization.
- Ensembling: Combining RDKit descriptors, hyperparameter optimization, and ensembling significantly improves performance on every dataset except MUV.Ensembling alone usually provides a small 1–5% benefit, with larger benefits on quantum-mechanics datasets.
- Data size: The D-MPNN may outperform Mayr et al.’s model at larger data thresholds, but the comparison is not fully conclusive.The authors hypothesize that D-MPNN struggles on low-label targets because it learns features from scratch without engineered descriptors.
Conclusion and Future Work
Across extensive public and proprietary benchmarks, the D-MPNN consistently matches or exceeds individual baselines and performs strongly in industrial datasets. Remaining weaknesses concern 3D information, very small datasets, and extreme class imbalance.
- Conclusion: D-MPNN achieves comparable or better performance than all baselines on 11 of 19 public datasets, while no single baseline dominates the remaining eight.The named public datasets include QM7, QM8, QM9, ESOL, FreeSolv, Lipophilicity, BBBP, PDBbind-F, PCBA, Tox21, and ClinTox.
- Conclusion: D-MPNN also shows strong performance on proprietary datasets, outperforming the random forest, feed-forward neural network, and Mayr et al. models on 15 of 16 datasets.The paper presents this as evidence that learned molecular representations can be used in industrial property prediction settings.
- Future work: The model typically underperforms when competing models use 3D information, the dataset is especially small, or the classes are particularly imbalanced.The examples given are quantum-mechanics and PDBbind models, PDBbind-C, and MUV, respectively.
- Future work: Future directions include adding richer 3D information, developing pretraining for limited-data settings, and adapting methods to extreme class imbalance.The paper also connects split-type analysis to future work on uncertainty quantification and domain-of-applicability assessment.
Supporting Information Available
The Supporting Information provides code, a web-based demonstration, additional split comparisons, raw performance statistics, class-balance analyses, and the RDKit feature list.
- Resources: The authors provide links to their code and a web-based user-interface demonstration.These resources are identified as freely available supporting materials.
- Additional analyses: Additional supporting analyses compare scaffold-based and random splits and provide raw chart performance numbers, including p-values.The Supporting Information also analyzes class balance in classification datasets.
- Model details: The Supporting Information lists the RDKit-calculated features used by the model.This complements the paper’s analysis of molecule-level feature additions.