PaperlayerView exact revision on arXiv

Source-linked AI summary

PotentialNet for Molecular Property Prediction

Evan N. Feinberg, Debnil Sur, Zhenqin Wu, Brooke E. Husic, Huanghao Mai, Yang Li, Saisai Sun, Jianyi Yang, Bharath Ramsundar, Vijay S. Pande

arXiv:1803.04465v2cs.LG

TL;DR

Molecular property prediction must handle limited high-quality data and interactions spanning different molecular length scales. The paper develops PotentialNet graph convolutions, evaluates them across structure-based and ligand-based tasks, and argues that clustered cross-validation and appropriate metrics better assess generalization. PotentialNet exceeds X-Score and RF-Score by over 0.5 average standard deviations on the agglomerative structure split, while performance depends on the validation split and metric.

  • Problem

    Molecular datasets are small relative to conventional deep-learning datasets, while standard evaluation can use similar train and test examples or encourage test-set hyperparameter overfitting.

  • Method

    The paper develops graph convolutions that incorporate bonded and noncovalent interactions, including staged updates, target-aware message functions, multitask networks, and agglomerative sequence or structure cross-validation.

  • Results

    PotentialNet exceeds X-Score and RF-Score by over 0.5 average standard deviations for EFχ^(R) on the agglomerative structure split, while ligand-based tasks show statistically significant performance increases.

  • Takeaways & Limitations

    Model generalization in structure-based drug discovery depends critically on both the cross-validation strategy and the held-out-test performance metric.

Abstract

from arXiv · show

The arc of drug discovery entails a multiparameter optimization problem spanning vast length scales. They key parameters range from solubility (angstroms) to protein-ligand binding (nanometers) to in vivo toxicity (meters). Through feature learning---instead of feature engineering---deep neural networks promise to outperform both traditional physics-based and knowledge-based machine learning models for predicting molecular properties pertinent to drug discovery. To this end, we present the PotentialNet family of graph convolutions. These models are specifically designed for and achieve state-of-the-art performance for protein-ligand binding affinity. We further validate these deep neural networks by setting new standards of performance in several ligand-based tasks. In parallel, we introduce a new metric, the Regression Enrichment Factor $EF_χ^{(R)}$, to measure the early enrichment of computational models for chemical data. Finally, we introduce a cross-validation strategy based on structural homology clustering that can more accurately measure model generalizability, which crucially distinguishes the aims of machine learning for drug discovery from standard machine learning tasks.

I. INTRODUCTION

Drug-like molecular prediction faces scarce scientific data and complex molecular structure, motivating graph-based neural architectures that learn from atoms, bonds, and broader interactions. The paper reviews graph convolutions and introduces models that propagate information through molecular structure and between molecules.

  • Motivation: Binding affinity is central to therapeutic ligand behavior, requiring high affinity for desired targets and low affinity for other macromolecules.
  • Motivation: Public molecular datasets are far smaller than conventional deep-learning datasets, creating a need for architectures suited to limited scientific data.
  • Motivation: 3D convolutions can impose high memory demands and exponentially growing parameter counts, producing a curse-of-dimensionality concern for volumetric molecular inputs.
  • Graph convolutional neural networks: Graph convolutions exploit molecular structure and permutation symmetry to use fewer parameters than orientation-specific 3D filters.
  • PotentialNet architectures: The paper's models extend graph convolutions to intramolecular and noncovalent interactions, including staged updates and distance- and target-feature-dependent messages.
  • Graph convolutional neural networks: A graph convolution updates each node using its own features and neighboring nodes, with separate message functions available for multiple edge types.
  • Graph convolutional neural networks: The reviewed GGNN uses GRU updates and edge-type-specific linear message functions shared across layers, propagating information to increasingly distant atoms.

3. Generalization to multitask settings

PotentialNet can generalize affinity prediction to multiple targets using either separate target-specific models or one multitask network. The multitask formulation uses a target-sized final weight matrix and averages binary cross-entropy across targets.

  • Affinity prediction across multiple targets can use separate models for each target or a single multitask network.
  • The multitask network's final weight matrix has one output dimension for each target in the dataset.Its shape is W (F CM) ∈ R(T ×fF CM−1), where T denotes the number of targets.
  • The multitask loss averages binary cross-entropy across the targets.

B. Structure-based scoring models

The paper extends graph convolutions to represent both intramolecular bonds and noncovalent interactions in protein-ligand systems. PotentialNet stages bonded atom-type derivation, spatial propagation, and ligand-only gathering within a unified framework.

  • Spatial graph convolutions: PotentialNet generalizes graph convolutions to multiple edge types, including covalent bonds and distance- or interaction-defined noncovalent contacts.The adjacency representation becomes a tensor whose edge-type dimension can encode bonds, spatial thresholds, and interactions such as hydrogen bonds or π−π stacking.
  • Staged PotentialNet: Stage 1 performs covalent-only propagation to derive vector-valued atom features from local bonded neighborhoods.These features are computed for both protein and ligand atoms before spatial information is introduced.
  • Staged PotentialNet: Stage 2 propagates information using both bonded and spatial interactions, allowing molecular features to incorporate protein-ligand proximity.The full adjacency tensor supplies the bond and non-bonded edge types used at this stage.
  • Staged PotentialNet: Stage 3 gathers features only over ligand atoms after they have incorporated bonded ligand information and spatial proximity to protein atoms.The block-matrix formulation makes this ligand-only gather computationally straightforward.
  • Unified representation: The framework derives atom types from shared initial features for protein and ligand atoms rather than using separate ligand and protein force fields.This provides a unified framework for deriving physicochemical properties of biomolecular interactions.

III. MEASURING EARLY ENRICHMENT IN REGRESSION SETTINGS FOR VIRTUAL SCREENING

Global regression metrics can be vulnerable to outliers and overlook the distribution and ranking priorities relevant to virtual screening. Early enrichment addresses the emphasis on identifying high-scoring molecules near the tail of the activity distribution.

  • Metric limitations: R2 and RMSE are susceptible to single-point outliers, while RMSE also ignores training-data distribution and null-model performance.AUC addresses the latter deficiency for classifiers, but not the regression setting described here.
  • Metric limitations: Global metrics weight all data points equally, although drug discovery typically prioritizes the top-scoring molecules selected for synthesis or purchase.Virtual screening may score millions of molecules but acts on only a small high-scoring subset.
  • Early enrichment: Early enrichment is the cheminformatics response to evaluating performance in the high-activity tail rather than across the entire dataset.The passage identifies early enrichment as an adopted concept for this virtual-screening objective.

A. Proposed Metric: EF (R)

The paper proposes EF_χ^(R), a regression early-enrichment metric based on the observed activities of the top χ% samples ranked by model predictions. Standard-deviation normalization enables comparisons across datasets with different measurement variances.

  • Motivation: EF_χ^(R) addresses the absence of an early-enrichment metric for regression, extending classification-oriented enrichment measurement to continuous outcomes.The metric is introduced specifically because existing early-enrichment progress had been limited to classification.
  • Definition: EF_χ^(R) ranks samples by predicted measurement and computes the average z-score of their observed measurements among the top χ%.The observed values are normalized by the dataset mean and standard deviation.
  • Normalization: Normalization by σ(y) allows model performance comparisons across datasets with different variances and gives the metric a common unit of measurement.Without normalization, the unnormalized score depends on the measurement distribution and shares the units of yi.
  • Interpretation: The upper bound depends on both the training-data distribution and the selected enrichment fraction χ.EF_χ^(R) values can exceed 1.0 when the top predicted molecules average more than one standard deviation above the mean.

IV. RESULTS

The paper addresses weaknesses in conventional molecular-model validation by introducing homology-based agglomerative cross-validation. PotentialNet achieves state-of-the-art performance on the standard PDBBind benchmark, but method rankings change under sequence- and structure-based splits.

  • Hyperparameter sensitivity and test-set searching make rigorous cross-validation critical for assessing molecular property models.
  • Cross-validation strategies: The standard refined-train/core-test split can overfit through hyperparameter searching and place structurally or sequentially similar examples in train and test sets.Removing training examples homologous to the core test set significantly attenuated reported affinity-prediction performance.
  • Cross-validation strategies: The proposed strategy divides data into train, validation, and test folds using agglomerative clustering based on protein sequence or structural similarity.Sequence-based clustering produced 978 train, 221 validation, and 101 test samples; structural clustering produced 925, 257, and 118 samples.
  • Benchmark performance: PotentialNet achieves state-of-the-art performance on the standard PDBBind 2007 refined-train/core-test benchmark and outperforms RF-Score and X-Score by Pearson and Spearman correlation.Its Pearson score is within error of the reported TopologyNet score, although the evaluation procedures differ.
  • Benchmark performance: On agglomerative splits, RF-Score performs best on sequence-based splitting, whereas PotentialNet performs best on structure-based splitting.The sequence split places Staged Spatial Graph Convolution and X-Score in a statistical tie behind RF-Score; the structure split places RF-Score second.
  • Benchmark performance: PotentialNet remains competitive despite using simple per-atom features and distance-based spatial convolutions, while competing methods incorporate domain-expertise-driven interaction information.

1. Sanity check with a traditional RNN

The paper tests whether its graph-based models outperform a sequential neural representation and evaluates them across ligand-based molecular-property tasks. PotentialNet improves performance on all three investigated ligand-based tasks, while staged spatial processing benefits QM8.

  • Sanity check with a traditional RNN: A bidirectional LSTM represents proteins by one-hot amino-acid sequences and ligands by character-level SMILES strings.
  • Sanity check with a traditional RNN: The LSTM reaches a test Pearson correlation coefficient of 0.518, substantially below the reported PotentialNet results.The coefficient corresponds to the model selected by the best validation score using the same metric.
  • Ligand-based models: QM8, ESOL, and Tox21 provide ligand-based benchmarks spanning three-dimensional electronic properties, solubility, and toxicity-related outputs.ESOL and Tox21 use two-dimensional molecular representations consisting of atoms and bonds.
  • Ligand-based models: PotentialNet yields statistically significant performance improvements on all three investigated ligand-based tasks.
  • Ligand-based models: QM8 experiments show that separating spatial graph convolutions into stages significantly improves performance.Stage I propagates only over bonds, while Stage II propagates over both bonded and distance-binned noncovalent interactions.

1. Quantum Property Prediction

PotentialNet was evaluated across quantum-property, toxicity, solubility, and protein-ligand affinity tasks. The models improved ligand-based performance and showed strong affinity prediction, while evaluation outcomes depended on the benchmark split and metric.

  • Quantum Property Prediction: Staged spatial PotentialNet achieved the best performances among evaluated models on the 12 QM8 quantum-property tasks.QM8 performance was measured by mean absolute error over 21,786 compounds.
  • Toxicity: PotentialNet achieved higher ROC-AUC scores than Weave and GraphConv on both validation and test sets in Tox21 toxicity prediction.The comparison used 100 epochs of training.
  • Solubility: PotentialNet achieved slightly smaller RMSE than Weave and GraphConv on the Delaney ESOL solubility task.The final test RMSE was comparable or superior to prior reported scores of 0.465 and 0.5216 for Weave and GraphConv, respectively.
  • Protein-Ligand Affinity: Spatial Graph Convolutions achieved state-of-the-art protein-ligand affinity performance using basic atom, bond, and distance inputs rather than hand-crafted interaction features.The framework excludes traditional features such as hydrophobic effects, π-stacking, and hydrogen bonding.
  • Protein-Ligand Affinity: On the traditional PDBBind 2007 benchmark, Spatial Graph Convolutions outperformed X-Score and RF-Score and performed comparably with TopologyNet.On agglomerative splits, sequence-based evaluation favored RF-Score, whereas structure-based evaluation favored Staged Spatial Graph Convolutions.
  • Evaluation Strategy: Regression Enrichment Factor EF_χ^(R) changed the affinity comparison across splits: PotentialNet lagged on sequence splits but exceeded X-Score and RF-Score on structure splits by over 0.5 average standard deviations.The structure-split result remained statistically tied with the ligand-only PotentialNet control.

METHODS

The study implemented PotentialNet and comparator models for affinity, electronic-property, toxicity, and solubility prediction. Training and evaluation used task-specific features, hyperparameter searches, and temporal or homology-based cross-validation procedures.

  • Models: PotentialNet models were implemented with PyTorch and molecular-processing code based on RDKit and OEChem.Random forest and linear regression comparators were implemented with scikit-learn.
  • Models: Affinity, electronic-property, toxicity, and solubility models used hyperparameters listed in Table VII, with toxicity and solubility restricted to bond graph convolutions.Those datasets did not provide 3D coordinates.
  • PDBBind 2007 Core Test: The affinity benchmark used temporal cross-validation on the PDBBind 2007 training set, selecting hyperparameters by average validation Pearson correlation.The test set consisted of the temporally ordered core set.
  • Agglomerative Clustering Splits: Sequence- and structure-based affinity splits used agglomerative clustering with Ward’s method and distances based on sequence homology or TM-score.Within each training set, random K-fold splits selected the model with the best average validation Pearson score.

SAFETY STATEMENT

The study reports no unexpected or unusually high safety hazards. The accompanying material identifies the sequence- and structure-based agglomerative clustering splits.

  • Safety Statement: No unexpected or unusually high safety hazards were encountered in the study.
  • Safety Statement: The listed supplementary material concerns sequence- and structure-based agglomerative clustering splits for the PDBBind 2007 refined set.

Appendix A: Computational complexity of network architectures

The appendix contrasts computational complexity across fully connected, convolutional, and graph-convolutional architectures. Graph message passing exploits sparsity and decomposed embeddings to reduce runtime for molecular graphs.

  • Fully Connected Networks: A fully connected network has total forward-propagation complexity O(n^3) when the number of layers h is much smaller than the input length n.Back-propagation has the same asymptotic complexity.
  • Convolutional Networks: A 2D convolution with an m × n filter over an M × N image requires O(MNmn) operations, or O(MN log MN) with FFT-based processing.The appendix notes that costs grow exponentially with the dimension of Euclidean data.
  • Graph Convolutions: Dense graph message passing requires O(n^2d^2) multiplications for n nodes with d-dimensional hidden representations.
  • Graph Convolutions: Breaking d-dimensional node embeddings into k components reduces the message-passing runtime to O(n^2d^2/k), typically O(nd^2/k) for sparse or small molecular graphs.The PotentialNet architectures build on this GGNN framework.
  • Benchmark Context: QM8 evaluation reports mean absolute error for DTNN, MPNN, and staged and single-update Spatial PotentialNet networks across all 12 tasks.
Loading 1803.04465v2…