PaperlayerView exact revision on arXiv

Source-linked AI summary

Massively Multitask Networks for Drug Discovery

Bharath Ramsundar, Steven Kearnes, Patrick Riley, Dale Webster, David Konerding, Vijay Pande

arXiv:1502.02072v1stat.MLcs.LGcs.NE

TL;DR

Drug discovery requires predictive models that can handle sparse, heterogeneous biological data. The paper trains massively multitask neural networks on large public datasets and empirically studies their performance, scaling, and transferability. Multitask models improve over simpler methods, benefit from more tasks and data, and transfer only limitedly to unseen tasks.

  • Problem

    Virtual screening has limited data for individual experiments, while effective prediction requires combining disparate experimental sources across targets.

  • Method

    The authors gather public experimental data and train massively multitask neural networks that share representations across biological targets.

  • Results

    Multitask networks significantly improve prediction, benefit from additional tasks and data, and provide limited transferability to tasks absent from training.

  • Takeaways & Limitations

    The results support greater sharing of experimental data and continued algorithmic innovation for drug discovery.

  • Takeaways & Limitations

    Overfitting is a major concern because some datasets contain only tens or hundreds of positive examples relative to many task-specific parameters.

Abstract

from arXiv · show

Massively multitask neural architectures provide a learning framework for drug discovery that synthesizes information from many distinct biological sources. To train these architectures at scale, we gather large amounts of data from public sources to create a dataset of nearly 40 million measurements across more than 200 biological targets. We investigate several aspects of the multitask framework by performing a series of empirical studies and obtain some interesting results: (1) massively multitask networks obtain predictive accuracies significantly better than single-task methods, (2) the predictive power of multitask networks improves as additional tasks and data are added, (3) the total amount of data and the total number of tasks both contribute significantly to multitask improvement, and (4) multitask networks afford limited transferability to tasks not in the training set. Our results underscore the need for greater data sharing and further algorithmic innovation to accelerate the drug discovery process.

1. Introduction

Drug discovery and virtual screening face expensive experiments, low hit rates, imbalanced data, and overfitting. The paper studies massively multitask networks as a way to combine data across targets and reports improved performance with more tasks and data.

  • Experimental drug discovery is lengthy, expensive, and prone to high failure rates while requiring efficacy, metabolic, and toxicity constraints.
  • Virtual screening uses computational models to replace or augment screening millions of compounds for promising molecules.
  • 1–2% of screened compounds are often active, creating imbalanced datasets vulnerable to artificial enrichment, analog bias, and overfitting.
  • Multitask networks combine experimental data across targets and share information to compensate for limited data in individual experiments.
  • Nearly 40 million measurements across over 200 targets were collected to study massively multitask learning for virtual screening.
  • Multitask performance improved with additional tasks and data, while transferability to unseen tasks was limited.

2. Related Works

Related work spans conventional machine learning, deep multitask models, and prior large-scale virtual-screening studies. The paper distinguishes its emphasis on explaining the sources of multitask improvement.

  • Earlier drug-discovery methods combined molecular featurizations with standard algorithms, neural networks, nearest neighbors, and Bayesian belief networks.
  • The Merck Kaggle competition used multitask deep neural networks among ensemble models and improved baseline test-set R2 by nearly 17%.
  • A contemporaneous study used 1,280 targets and 2 million data points, whereas this work used 259 tasks and nearly 40 million data points.
  • This paper focuses on disentangling the underlying causes of multitask improvements rather than only highlighting performance gains.
  • Multitask deep networks had also been applied to language understanding and multilanguage speech recognition.

3. Methods

The study trains multitask neural networks on diverse public bioassay datasets using molecular fingerprints, stratified cross-validation, and ROC-based metrics. Its architecture shares representations while attaching one classifier to each dataset.

  • Dataset construction: 259 public datasets from PCBA, MUV, DUD-E, and Tox21 were used for model training.
  • Dataset construction: The datasets were not generally preprocessed to remove experimental artifacts, although MUV was designed to address such pathologies.
  • Molecular representation: ECFP4 fingerprints represent molecules as hashed fixed-length bit vectors built from atom-centered radial fragments.
  • Validation and metrics: Stratified 5-fold cross-validation preserved active/inactive proportions in every fold to address variability from imbalanced datasets.
  • Validation and metrics: Because no explicit validation set was used, hyperparameter selection may have experienced information leakage across folds.
  • Validation and metrics: The study used mean and median fold-averaged AUC derived from ROC curves to evaluate collections of datasets.
  • Network architecture: A neural network repeatedly applies learned linear transformations and nonlinearities, then feeds the final representation to a classifier.
  • Network architecture: Multitask networks attach one softmax classifier per dataset to a shared final layer.

4. Experimental Section

The experimental section asks whether massively multitask networks improve performance, how task and data quantity affect results, and whether learned chemical information transfers to unseen tasks.

  • The experiments test whether massively multitask networks outperform simple machine-learning methods and which architecture is optimal.
  • The study examines how multitask performance depends on the number of tasks and the total amount of data.
  • The experiments assess whether massively multitask networks extract information about chemical space that generalizes to new tasks.

4. When do datasets benefit from multitask training?

The experiments compare multitask architectures with standard and single-task baselines, then examine how performance changes as tasks and data are added. Results favor pyramidal multitask networks and suggest continued gains, although overfitting and eventual growth-curve behavior remain important qualifications.

  • Model comparisons: Pyramidal multitask networks significantly outperformed simpler models, including the best single-task baseline.The baseline is defined as Max{LR, RF, STNN, PSTNN}.
  • Model comparisons: DUD-E models all achieved median 5-fold-average AUCs of at least 0.99, making comparisons on those datasets uninformative.DUD-E was excluded from subsequent statistical analysis but retained during training because removing it adversely affected other datasets.
  • Architecture and training: Overfitting was a major issue when each dataset had 1200 associated parameters but only tens or hundreds of positive examples.The issue arose in the absence of strong regularization.
  • Architecture and training: Pyramidal networks reduce dataset-specific parameters through a wide 2000-node layer followed by a narrow 100-node layer, with dropout of 0.25 improving performance.The narrow layer limits task-specific parameters while the wide layer supports expressive feature learning.
  • Architecture and training: Across pyramidal layer sizes, means and medians shifted by ≤.01 AUC, except MUV, which showed a range of .038.Performance was also sensitive to learning rate and the number of training steps.
  • Task and data growth: Held-in performance was evaluated while expanding nested multitask collections from 10 to 249 tasks, with additional tasks selected randomly.The experiment used fixed held-in tasks and repeated the procedure with ten random seeds.
  • Task and data growth: All held-in datasets showed subsequent improvement after some initial dips, and all but one exceeded the single-task baseline.The mean held-in performance was still increasing at 249 tasks, consistent with either a plateau or a still-climbing curve within the collection.

4.3. More tasks or more data?

The experiments separate the contributions of task count and data volume to multitask improvement. Both factors consistently improve performance, while transfer to held-out tasks is limited and strengthens with larger multitask training sets.

  • Task and data contributions: More tasks consistently improved held-in performance when the total amount of additional data was fixed.Figure 4 varied the number of additional tasks while controlling total additional input examples.
  • Task and data contributions: Additional data consistently improved performance when the number of tasks was fixed.The experiment independently varied sampled data volume and task count.
  • Task and data contributions: The total amount of data and total number of tasks both contributed significantly to the multitask effect.The study compared networks containing 10, 15, 20, 30, 50, or 82 tasks and sampled 1.6M to 23M non-held-in data points.
  • Held-out transfer: Transfer learning to held-out datasets could be negative, with many datasets performing worse than the single-task baseline after multitask initialization.Held-out networks were initialized from multitask weights and then fine-tuned using 5-fold stratified cross-validation.
  • Held-out transfer: Transfer effects strengthened with more multitask training data, but average transferability remained only ∼.01 AUC even with 249 datasets.The authors hypothesize that generalizability depends on whether relevant data were present in multitask training.

4.5. When do datasets benefit from multitask training?

The authors examine why multitask benefits vary across datasets by relating improvement to shared active compounds, target classes, and duplicate targets. Benefits occur across nearly every target class, correlate moderately with active-compound overlap, and are unlikely to reflect substantial information leakage from duplicate targets.

  • Shared active compounds: AOR measures dataset similarity by counting additional datasets in which each active compound is also active.For dataset Di and active compound α, AOR_i,α is the number of additional datasets sharing that active compound.
  • Shared active compounds: A moderate correlation linked AOR with multitask improvement in log-odds-mean-AUC, with r2 = .33.The correlation was not present when improvement was measured as ∆mean-AUC, for which r2 = .09.
  • Target classes: Nearly every target class realized multitask gains, with no class benefiting qualitatively more than another.DUD-E datasets were excluded, and classes with fewer than five members were merged into “miscellaneous.”
  • Duplicate targets: Duplicated targets had slightly higher average log-odds improvement than unique targets, .531 versus .372, with p = .016.The duplicate and unique distributions substantially overlapped.
  • Duplicate targets: Overlapping confidence intervals and sign tests suggest duplicate targets did not produce significant information leakage in the multitask networks.The reported intervals were [0.04, 0.24] for duplicate targets and [0.06, 0.17] for unique targets.

5. Discussion and Conclusion

Massively multitask neural networks improved virtual screening and benefited from additional tasks and data, while transfer to unseen tasks remained limited. The authors emphasize data sharing, benchmark standardization, improved featurization, and the continuing complexity of drug discovery.

  • Massively multitask neural networks achieved significant improvement over simple machine learning algorithms for virtual screening.
  • Performance was still climbing at 259 tasks, while additional data and additional tasks both contributed significantly to the multitask effect.The effect diminished as more data and tasks were added but did not appear to plateau within the collection.
  • Transferability to tasks outside the training set was limited, not universal, and required large amounts of data when it applied.
  • Shared active compounds were moderately correlated with multitask improvement, whereas the biological class of the target was not.The authors also reported strong improvement on the thoroughly cleaned MUV datasets, arguing against an explanation based only on experimental artifacts.
  • The efficacy of multitask learning was directly related to relevant-data availability, supporting increased sharing of private experimental measurements.
  • Published deep-learning applications used distinct, non-comparable datasets, leaving standard datasets and performance metrics for future research.
  • The study used only ECFP4 featurization, leaving target-aware featurization and unsupervised exploration of chemical space as future directions.
  • Although deep learning offers possibilities for virtual screening, the full drug discovery process remains immensely complicated.

A. Dataset Construction and Design

The dataset collection combines diverse public assay sources spanning 259 datasets and multiple target classes. Its datasets differ in overlap patterns, target duplication, and assay composition, with construction details documented in supplementary tables and figures.

  • PCBA datasets were dose-response assays downloaded from PubChem BioAssay using specified activity-count, chemical, assay-type, and target filters.
  • The collection contains 259 datasets spanning a wide range of target classes and assay types, including cell-based and in vitro experiments.Duplicated targets are marked with an asterisk; non-DUD-E duplicate-target datasets were used in the text analysis.
  • Figure A.2 encodes each dataset pair’s intersection as the fraction of dataset x contained in dataset y, with black lines separating dataset groups.
  • DUD-E datasets have very little intersection with any other datasets, while Tox21 datasets have very small intersections with other groups.
  • PCBA and Tox21 datasets have substantial self-overlap, whereas MUV datasets have relatively little self-overlap.
  • MUV datasets have substantial overlap with datasets in the PCBA group.
  • Figure A.3 compares ∆log-odds-mean-AUC distributions for datasets with duplicate and unique targets.
  • The supplementary materials include tables for featurization failures, held-in and held-out datasets, target-class breakdowns, and dataset intersections.

B. Performance metrics

Performance is evaluated with group-level statistical comparisons and enrichment measures. The reported comparisons use the PMTNN as the reference across non-DUD-E datasets, with boxplot uncertainty and paired tests providing additional context.

  • Table B.1 compares each model with the Pyramidal (2000, 100) Multitask Neural Net using sign-test confidence intervals for each dataset group.
  • Enrichment measures how much better than random a model’s top X% predictions perform, using the ROC enrichment definition.

C. Training Details

Training used stochastic gradient descent with specified initialization, learning-rate, batch-size, and step schedules across multitask, single-task, and sensitivity-analysis models. Class weighting addressed the strong imbalance between inactive and active compounds.

  • Multitask networks used learning rate .0003, batch size 128, 50M steps, Gaussian weight initialization with standard deviation .01, and bias initialization at .5.Higher learning rates sometimes caused pyramidal networks to fail when the top hidden layer zeroed itself out.
  • Pyramidal single-task networks used the same settings for 100K steps, while vanilla single-task networks used learning rate .001 for 100K steps.
  • Figure 5 networks used a Pyramidal (1000, 50) Single Task architecture initialized from Figure 3 weights and trained for 100K steps at learning rate 0.0003.
  • Active compounds were weighted so their total weight equaled the number of inactives, which retained unit weight.
  • Tables C.1–C.3 report pyramid sensitivity results and additional models, including median 5-fold-average-AUC values relative to the PMTNN.
  • The pyramid sensitivity analysis used learning rate 0.0001 for the first 2M steps and 0.0003 for the next 28M steps.
  • Additional-model descriptions define MTNN as multitask neural net and auxiliary heads as independent softmax units attached to hidden layers.
Loading 1502.02072v1…