Source-linked AI summary
Low Data Drug Discovery with One-shot Learning
Han Altae-Tran, Bharath Ramsundar, Aneesh S. Pappu, Vijay Pande
TL;DR
Drug-discovery machine learning is constrained by limited compound data, despite deep networks' ability to improve molecular prediction. The paper introduces one-shot learning with residual LSTM and graph-convolutional architectures, reporting strong boosts over simpler methods on Tox21 and SIDER while noting limited generalization across diverse scaffolds.
Problem
Drug-discovery campaigns often have too little biological data to characterize compounds or support accurate predictions for novel molecules.
Method
The paper adapts one-shot learning to drug discovery and introduces residual LSTM models with graph-convolutional molecular representations.
Results
One-shot learning methods strongly dominate simpler machine-learning baselines on the Tox21 and SIDER collections.
Takeaways & Limitations
The work establishes low-data learning as a drug-discovery task and provides an architecture for learning such models.
Takeaways & Limitations
The authors report strong limitations to generalization because of diverse scaffolds in the MUV collection.
Abstract
from arXiv · showhide
Recent advances in machine learning have made significant contributions to drug discovery. Deep neural networks in particular have been demonstrated to provide significant boosts in predictive power when inferring the properties and activities of small-molecule compounds. However, the applicability of these techniques has been limited by the requirement for large amounts of training data. In this work, we demonstrate how one-shot learning can be used to significantly lower the amounts of data required to make meaningful predictions in drug discovery applications. We introduce a new architecture, the residual LSTM embedding, that, when combined with graph convolutional neural networks, significantly improves the ability to learn meaningful distance metrics over small-molecules. We open source all models introduced in this work as part of DeepChem, an open-source framework for deep-learning in drug discovery.
Introduction
Deep-learning methods offer powerful molecular representations but usually require more data than drug-discovery campaigns can provide. This work adapts one-shot learning and introduces residual LSTM models to improve low-data molecular prediction.
- Motivation: Deep neural networks learn sophisticated input representations, but their predictive capability depends on large amounts of training data.Massively multitask networks have been trained with millions of datapoints.
- Motivation: Drug-discovery pipelines often struggle to characterize even a few dozen compounds, while even hundreds can be too resource intensive.Limited biological data makes accurate prediction for novel compounds challenging.
- One-shot learning: One-shot learning uses related data to learn a distance metric that compares new datapoints with limited available data to predict their properties.These methods are designed to learn non-trivial predictors from only a few datapoints.
- One-shot learning: The paper adapts one-shot learning from recognizing new classes to predicting molecule behavior in new experimental systems.The stated drug-discovery challenge is to predict molecular behavior rather than recognize a new class from one example.
- Architecture: The residual LSTM is introduced as a modification of matching networks and residual convolutional networks for learning metrics that trade information between evidence and query molecules.It is combined with graph-convolutional processing of small molecules, whose learnable features can be modified to improve performance.
- Results and release: One-shot methods strongly dominate simpler machine-learning baselines on the Tox21 and SIDER collections.The models and graph-convolutional primitives are open-sourced through DeepChem, with scripts and datasets provided for reproducing the results.
Methods
The method adapts one-shot learning to binary drug-discovery tasks with limited data by learning distance-based predictions from support sets. It combines graph-based molecular representations with context-aware and iteratively refined embeddings.
- One-shot learning setup: The model reserves some binary learning tasks for training and uses their information to classify test systems with too little data for standard models.Each task contains labeled small-molecule examples, and the target is a classifier conditioned on an available support set.
- One-shot learning setup: One-shot prediction uses attention-weighted combinations of support-set labels, where attention reflects similarity between query and support embeddings.The final prediction is interpreted as the expected support-label value under the attention distribution.
- Graph-based molecular representations: The architecture replaces image-style encoders with graph-convolutional networks that embed small molecules into a continuous representation space.The query and support embeddings are compared with a similarity measure such as cosine distance.
- Context-aware embeddings: Matching-network embeddings provide full context by allowing query and support representations to affect one another, which empirically gives stronger one-shot results.Earlier independent embeddings leave the query representation without information from the support set.
- Iterative refinement using dual residual LSTMs: The residual LSTM architecture iteratively updates query and support embeddings together, resolving support-set order dependence and asymmetric treatment of query and support.An attention LSTM generates updates that are added to the initial embeddings before one-shot prediction.
- Graph operations: Graph pooling takes the maximum activation across a node and its neighbors, while graph-gather sums node features into one fixed-size graph descriptor.These operations expand downstream receptive fields without increasing parameters and produce a graph-level vector for prediction.
Experiments
Across Tox21 and SIDER, one-shot learning methods outperform the random-forest baseline, with Residual LSTM models showing more robust gains when data are limited. On MUV, however, the random forest performs better and Residual LSTM does not achieve the best performance; cross-collection transfer from Tox21 to SIDER yields no predictive power.
- Tox21 and SIDER: One-shot learning methods show strong boosts over the random-forest baseline on both Tox21 and SIDER held-out tasks.The reported table values are median test-task accuracies, averaged across 20 random support-set choices per task.
- Tox21 and SIDER: Residual LSTM models provide more robust boosts than the random-forest baseline in the presence of less data on Tox21 and SIDER.This pattern is reported for both benchmark collections.
- MUV: MUV is a difficult low-data setting because structurally distinct positive examples limit the ability to exploit similarity when predicting new active compounds.The collection is described as favorable for baseline machine learning but unfavorable for low-data methods.
- MUV: On MUV, the random-forest baseline performs significantly better than on the other tested datasets, while Residual LSTM does not achieve the best performance.The authors suggest that one-shot learning methods may have difficulty generalizing to new molecular scaffolds.
Discussion and Conclusion
The paper introduces low-data learning for drug discovery and reports strong performance on Tox21 and SIDER, while identifying important limits to generalization across diverse scaffolds and unrelated tasks.
- The paper introduces low data learning for drug discovery and provides an architecture for learning such models.
- One-shot learning methods strongly dominate simpler machine learning baselines on the Tox21 and SIDER collections.
- The SIDER result is notable because the collection contains high-level phenotypic side-effect observations with substantial uncertainty.
- These results indicate that one-shot learning methods could perform strongly on small biological datasets, including a small number of drug tests in rats.
- One-shot learning methods struggle against simple baselines on MUV, likely because its diverse scaffolds challenge generalization to novel molecular scaffolds.
- Transfer learning from Tox21 to SIDER causes all one-shot methods to collapse, revealing limited cross-task generalization whose precise limits remain for future work.
Definitions of Graph Primitives
The molecular graph is featurized through graph convolution, pooling, and gathering operations applied across nodes. The graph convolution uses degree-dependent parameters, a bias, and a nonlinearity.
- Graph convolution, graph pooling, and graph gathering are the three neural-network layers used to featurize molecular graphs.
- Definitions given for one node are applied simultaneously to all nodes when operating on the full graph.The convolution output is represented as a sequence containing the outputs for each node, with pooling treated similarly.
- The graph-convolution expression combines degree-dependent node and neighborhood terms with a degree-dependent bias.
- The graph convolution uses a nonlinearity such as ReLU or tanh.
Convolutional Architecture in this Paper
The Siamese, AttnLSTM, and ResLSTM models use the same convolutional architecture. Inputs feed sequentially through the listed layers from left to right.
- The Siamese, AttnLSTM, and ResLSTM models all use the same convolutional architecture.
- The architecture table depicts input entering on the left and feeding sequentially through layers toward the right.
- Table 5 is identified as the Convolutional Network Architecture.
Molecular Features
Molecules are represented as undirected graphs with atoms as nodes and bonds as edges. RDKit-derived atomic features provide the initial inputs to graph-convolutional layers.
- Each molecule is featurized as an undirected graph whose nodes are atoms and whose edges are bonds.
- Bond types are not distinguished in the molecular graph representation.
- RDKit computes atom-type, valence, formal-charge, and hybridization features for each atom.
- These atomic features form the initial feature set supplied to graph-convolutional layers.
Tox21 Details
The Tox21 setup uses nine assays for training and three separate assays for model evaluation.
- Nine assays—NR-AR, NR-AR-LBD, NR-AhR, NR-Aromatase, NR-ER, NR-ER-LBD, NR-PPAR-gamma, SR-ARE, and SR-ATAD5—were used for training.
- Three assays—SR-HSE, SR-MMP, and SR-p53—were used for model evaluation.
SIDER Details
The SIDER details identify indications used for training and a separate set used for model evaluation. Training covered numerous disorder categories, while evaluation included renal and urinary disorders.
- Training used indications spanning hepatobiliary, metabolic, pregnancy-related, ear, cardiac, nervous-system, injury-related, eye, musculoskeletal, gastrointestinal, social, immune, reproductive, neoplastic, general, endocrine, surgical, vascular, blood, skin, congenital, infectious, respiratory, and psychiatric disorders.
- The listed SIDER indications were divided between training and model evaluation.
- Evaluation used indications including renal and urinary disorders.
MUV Details
The MUV details distinguish assays used for training from assays reserved for model evaluation. Training included twelve named assays, whereas evaluation included five different assays.
- Twelve assays—MUV-466, MUV-548, MUV-600, MUV-644, MUV-652, MUV-689, MUV-692, MUV-712, MUV-713, MUV-733, MUV-737, and MUV-810—were used for training.
- The MUV assay split assigned distinct assay sets to training and model evaluation.
- Five assays—MUV-832, MUV-846, MUV-852, MUV-858, and MUV-859—were used for model evaluation.