Source-linked AI summary
Multi-task Neural Networks for QSAR Predictions
George E. Dahl, Navdeep Jaitly, Ruslan Salakhutdinov
TL;DR
QSAR research has been dominated by methods such as random forests, while neural-network approaches have renewed interest through newer techniques. This paper applies multi-task neural networks with modern regularization to predict compound activities across assays, and reports superior performance to alternative baselines. The authors also identify limits related to dataset size, task formulation, and descriptor coverage.
Problem
QSAR needs accurate models relating chemical structure to compound properties or activities, while recent literature has often favored methods such as random forests over neural networks.
Method
The paper trains neural networks that predict multiple assays simultaneously, using shared learning and recent techniques for controlling overfitting.
Results
Neural-network methods provide superior performance, with the best neural net significantly exceeding the best baseline on 14 of 19 assays and showing no significant difference on the remaining five.
Takeaways & Limitations
Multi-task neural networks offer an effective way to leverage data from multiple assays for QSAR prediction.
Takeaways & Limitations
The experiments use binary classification with potentially unreliable active/inactive labels, and the descriptor set may omit features available in other software.
Abstract
from arXiv · showhide
Although artificial neural networks have occasionally been used for Quantitative Structure-Activity/Property Relationship (QSAR/QSPR) studies in the past, the literature has of late been dominated by other machine learning techniques such as random forests. However, a variety of new neural net techniques along with successful applications in other domains have renewed interest in network approaches. In this work, inspired by the winning team's use of neural networks in a recent QSAR competition, we used an artificial neural network to learn a function that predicts activities of compounds for multiple assays at the same time. We conducted experiments leveraging recent methods for dealing with overfitting in neural networks as well as other tricks from the neural networks literature. We compared our methods to alternative methods reported to perform well on these tasks and found that our neural net methods provided superior performance.
1 Introduction
The paper revisits neural networks for QSAR by combining multi-task learning with modern neural-network techniques for controlling overfitting. It evaluates whether shared modeling across assays can improve prediction over established approaches.
- Motivation: QSAR models relate compounds’ chemical structures to properties or activities, supporting in silico testing and ranking without wet-lab experiments.QSAR is already used for properties including absorption, distribution, metabolism, excretion, and toxicity.
- Prior work: QSAR machine learning has included linear regression, Bayesian neural networks, random forests, projection pursuit, partial least squares, and support vector machines.Practitioners also value variable selection, uncertainty assessment, and control of overfitting on small datasets.
- Contribution: The paper’s central novelty is multi-task neural networks that operate on multiple assays simultaneously, a relatively rare approach in QSAR.The authors also use dropout and do not rely on target-protein features.
- Motivation: Multi-task learning shares feature extraction and statistical strength across related assays, potentially producing more general features from molecular descriptors.QSAR assays are natural multi-task targets because assays and compounds may share chemically relevant features.
- Contribution: The experiments apply multi-task learning with neural networks and report significantly improved results over random-forest baselines.The paper combines this approach with recent developments for neural-network training and regularization.
2 Methods
The methods use feedforward neural networks to transform molecular descriptors into assay predictions, with multi-task outputs allowing shared learning across assays. The study also addresses limited QSAR data through regularization and training procedures designed for high-capacity networks.
- Neural-network model: Feedforward neural networks map input vectors to output vectors through layered nonlinear transformations that learn task-relevant features.The internal layers re-represent the inputs while extracting useful features.
- Neural-network model: An L-layer network is parameterized by weight matrices and bias vectors, with each layer applying an elementwise nonlinear activation to its net input.The activations define the successive representations used to produce the output.
- Objectives: The network parameters are optimized by minimizing a cost function, using mean squared error for standard regression problems and cross-entropy for classification.The paper gives separate objective functions for regression and binary classification.
- Optimization: Training uses minibatched stochastic gradient descent with momentum, repeatedly estimating parameter gradients from small batches and updating parameter velocity.The update rule is described as operating on minibatches of training cases.
- Multi-task neural networks: The multi-task architecture predicts multiple assays simultaneously from molecular descriptors using a separate output unit for each assay.For compounds with partially observed assay outcomes, backpropagation updates only weights connected to observed outputs.
- Multi-task neural networks: Multi-task training controls each assay’s contribution to minibatches so assays with more compounds do not dominate the objective.A compound appearing in k assays is treated as k training cases sharing one descriptor vector but having different observed outputs.
- Regularization: Dropout regularizes wide and deep networks by randomly zeroing neuron activations during training, helping address overfitting when QSAR data are limited.The paper presents dropout as one of several regularization approaches considered for high-capacity models.
3 Experiments
The experiments evaluated multi-assay QSAR classification using PubChem data, molecular descriptors, held-out test sets, baseline classifiers, and Bayesian-optimized neural-network metaparameters.
- Data: 19 PubChem assays, including cellular and biochemical assays and related assay families, supplied the experimental data.
- Data: 3764 Dragon3 molecular descriptors remained after excluding descriptors inapplicable to all compounds and were Z-score normalized across the assay union.
- Task formulation: Binary active/inactive labels defined the classification task, although QSAR prediction can also be formulated as regression or ranking.
- Evaluation: 25% of ligands were held out per assay for testing, while random forests, gradient boosted decision trees, and logistic regression served as baselines.
- Neural-network tuning: Neural-network metaparameters covering architecture, optimization, and regularization were selected by Bayesian optimization of validation AUC.
- Neural-network tuning: Bayesian optimization efficiently found configurations but could overfit small validation sets, potentially reducing neural-network test performance relative to simpler-to-tune baselines.
4 Results and discussion
Across 19 assays, multi-task neural networks generally outperformed decision-tree and single-task baselines, especially when related assays provided useful information. Results also show that regularization enabled broad descriptor sets, while optimal network depth varied by assay.
- Multi-tasking vs baselines: On 14 of 19 assays, the best neural net achieved a statistically significant test AUC advantage over the best baseline; the other five showed no significant difference.The baselines were random forests and gradient-boosted decision-tree ensembles; logistic regression performed worst across all assays.
- Multi-tasking vs combining assays: On 8 of 11 combined-assay datasets, multi-task neural nets significantly outperformed both single-task GBMs and GBMs trained on combined related-assay data.On the remaining three datasets, the best and second-best models did not differ significantly.
- Multi-tasking vs combining assays: For related assays, multi-task networks improved over single-task networks on all but two of 11 assays, often without prior knowledge of assay relationships.The models were trained jointly on all 19 assays and could learn to ignore irrelevant tasks.
- Multi-tasking vs combining assays: The benefit of multi-task learning depended on assay relationships: combined GBM training helped the closely related 48891* series, whereas multi-task networks handled less directly aligned relationships more effectively.On the 1851* series, combined-training GBMs performed worse than single-task GBMs, while multi-task networks improved over both; on 46321*, multi-task networks were better still than combined GBMs.
- Controlling overfitting without feature selection: Well-regularized neural networks handled thousands of correlated descriptors, with 2000–2500 informative features usually preserving AUC while 1500 or fewer typically caused a large unnecessary drop.Bayesian optimization selected dropout and non-zero L2 penalties, and the best network for every assay used dropout.
- Neural network depth: Changing hidden-layer count had no consistent overall effect, although deeper multi-task models were occasionally important and assay-specific optimal depth was unpredictable.The paper notes that its depth findings differed from the authors’ experience in a non-public QSAR competition and that alternative descriptor sets might improve results.
5 Conclusions and future work
The results show that modern neural-network techniques improve QSAR prediction accuracy, with multi-task learning providing an effective way to use data from multiple assays. The authors identify several directions for further methodological development.
- Neural networks using recent deep-learning techniques improved QSAR prediction accuracies.
- Multi-task neural networks provide a natural and effective way to leverage data from multiple assays during QSAR training.
- Further experiments are needed to determine the best neural-network approach for QSAR problems.
- The authors plan to study ranking-based formulations because binary active/inactive labels from PubChem assay depositors may be unreliable for virtual screening.
A Stochastic gradient descent details
Neural-network training used minibatch stochastic gradient descent with momentum and backpropagation. The weight updates combine the minibatch gradient, momentum, learning rate, and weight-cost regularization.
- Training used minibatch stochastic gradient descent with momentum and backpropagation to compute gradients.
- The update formulas use the average objective-function gradient over the current minibatch.
- ϵ is the learning rate, α is the momentum strength, and λ is the weight-cost strength.
B Bayesian optimization search space
Bayesian optimization searched a broad neural-network hyperparameter space using constrained Spearmint, including regularization, architecture, initialization, dropout, training duration, and learning-rate schedules.
- Constrained Spearmint optimized neural-network metaparameters with a usual budget of 30 sequential trials and treated divergent runs as constraint violations.
- The search varied dropout fractions from 0 to 0.75 separately for the input and each hidden layer.
- The number of training epochs ranged from 2 to 100 for one-hidden-layer nets and from 2 to 120 for deeper nets.
- The optimization selected the number of hidden units in each layer, subject to architecture-specific minimum and maximum sizes.Multi-task networks used at least 512 units per hidden layer and up to 3584, except three-hidden-layer models capped at 2048.
- The search included annealing type, L2 weight cost, activation function, initial-weight scale, input-layer weight scaling, annealing delay, and initial learning rate.Annealing could be exponential or linear; activation functions could be logistic sigmoids or rectified linear units.
C Statistical signifigance determination
The study used bootstrap sampling to assess whether differences in test AUC between models were large relative to variability across training samples. The procedure was designed for comparisons across assays with different data sizes.
- The procedure compared model test AUC differences with uncertainty estimated from bootstrap variances.
- Bootstrap sampling provided standard errors for significance tests by retraining models on 8 bootstrap samples of the complete training set.
- The approximate test used cross-validation AUC differences while accounting for the performance degradation caused by training on bootstrap samples.
- For GBMs trained on combined assay data, reusing single-task GBM standard errors may have overestimated uncertainty and favored those GBM comparisons.