Source-linked AI summary
Self-Attention Based Molecule Representation for Predicting Drug-Target Interaction
Bonggun Shin, Sungsoo Park, Keunsoo Kang, Joyce C. Ho
TL;DR
DTI prediction is costly and difficult, creating a need for computational methods that can accelerate drug discovery. The paper introduces a self-attention-based molecule representation pre-trained on chemical compounds and transfers it to MT-DTI. MT-DTI outperforms existing methods on benchmark evaluations and ranks known EGFR drugs among the top-30 candidates.
Problem
DTI prediction is time-consuming and expensive, while existing CNN and one-hot representations have limitations in modeling distant atom relationships and using chemical structure knowledge.
Method
MT-DTI pre-trains a self-attention-based molecule representation on PubChem compounds, then transfers and fine-tunes it for DTI prediction.
Results
MT-DTI outperforms the state-of-the-art model by 4.9% points on Kiba and 1.6% points on Davis in area under the precision-recall curve.
Takeaways & Limitations
The model ranks all existing EGFR drugs among the top-30 candidates, suggesting potential usefulness for lower-cost drug discovery and personalized medicine.
Takeaways & Limitations
The paper identifies CNNs’ limited ability to model distant atom relationships and one-hot encoding’s failure to exploit chemical structure knowledge as motivating limitations of prior representations.
Abstract
from arXiv · showhide
Predicting drug-target interactions (DTI) is an essential part of the drug discovery process, which is an expensive process in terms of time and cost. Therefore, reducing DTI cost could lead to reduced healthcare costs for a patient. In addition, a precisely learned molecule representation in a DTI model could contribute to developing personalized medicine, which will help many patient cohorts. In this paper, we propose a new molecule representation based on the self-attention mechanism, and a new DTI model using our molecule representation. The experiments show that our DTI model outperforms the state of the art by up to 4.9% points in terms of area under the precision-recall curve. Moreover, a study using the DrugBank database proves that our model effectively lists all known drugs targeting a specific cancer biomarker in the top-30 candidate list.
1. Introduction
DTI prediction is costly and difficult, motivating scalable in silico methods that can support drug discovery and personalized medicine. The paper proposes MT-DTI, which combines self-attention-based molecule representation with pre-training on chemical compounds and reports improved benchmark performance.
- Motivation: DTI prediction is essential to drug discovery, but experimental assays are time-consuming, expensive, and yield fewer than 10% accepted new drugs.In silico prediction can suggest candidate molecules promptly and reduce overall process costs by up to 43%.
- Limitations of prior methods: Similarity-based methods limit molecule representation to similarity space and require similarity matrices, hindering new-molecule prediction and training scale.These limitations can ignore information embedded in molecule sequences and make predictions inaccurate for dissimilar molecules.
- Limitations of prior methods: CNN-based molecule encoders cannot model potential relationships among distant atoms in raw molecule sequences.The paper motivates self-attention as a mechanism capable of capturing relationships among atoms across the sequence.
- Representation learning: One-hot molecule encoding does not exploit existing chemical structure knowledge, motivating pre-training on compounds from PubChem.The paper proposes transferring this learned representation to DTI prediction through fine-tuning.
- Proposed approach: MT-DTI uses a self-attention-based molecule representation pre-trained on 97 million PubChem molecules and fine-tuned for DTI prediction.The model is evaluated on the Kiba and Davis benchmark datasets using molecule and protein sequence inputs.
- Results: 4.9% points for Kiba and 1.6% points for Davis are the reported AUPR improvements over the current state-of-the-art model.The paper also reports that the model ranks all known FDA-approved drugs targeting a specific protein highly in candidate lists.
2. Methods
The method combines a new molecule-representation approach with a DTI architecture designed to capture long-distance relationships in molecule sequences.
- 2. Methods: Molecule Transformers are introduced to model contextual and structural information among atoms in molecule sequences.
- 2. Methods: MT-DTI uses self-attention to address the inability of CNN-based molecule representations to relate distant atoms.The proposed architecture includes Molecule Transformers as a central building block.
- 2. Methods: The proposed method consists of Molecule Transformers, Protein CNNs, and Interaction Denses.
2.1. Model Architecture
MT-DTI processes SMILES molecule sequences and FASTA protein sequences, then combines their learned encodings for interaction prediction.
- 2.1. Model Architecture: The model encodes a SMILES molecule with Molecule Transformers and a FASTA protein with Protein CNNs.
- 2.1. Model Architecture: The resulting molecule and protein encodings are jointly supplied to Interaction Denses for DTI prediction.
- 2.1. Model Architecture: The molecule architecture uses token and positional embeddings to construct model inputs for molecule sequences.Figure 5 illustrates this process for methyl isocyanate (CN=C=O).
2.2. Molecule Transformers
Molecule Transformers use bidirectional self-attention with token and positional embeddings to encode SMILES sequences, and are pretrained by predicting masked tokens.
- 2.2. Molecule Transformers: Molecule Transformers are bidirectional Transformer encoders that model relationships among long-distance atoms without recurrent or convolutional sequence layers.
- 2.2.1. Input Embedding: The input representation sums trainable molecule token embeddings with trainable positional embeddings.The positional component supplies sequence-order information that self-attention does not otherwise consider.
- 2.2.1. Input Embedding: Special tokens support padding, representation extraction, sequence boundaries, and handling sequences longer than the maximum length.
- 2.2.2. Self-Attention Layer: Self-attention projects molecule inputs into query, key, and value representations, allowing learned atom relationships to span the entire sequence.The key dimension is D_k.
- 2.2.3. Multi-Head Attention: Multi-head attention concatenates multiple attention outputs and projects them into the Transformer's final output representation.
- 2.2.4. Pre-training: Pretraining masks SMILES tokens and trains the Transformer to predict the original token, while the final [REP] vector becomes the fixed-length molecule representation.The pretrained weights initialize the Molecule Transformers used in MT-DTI.
2.3. Protein CNNs
Protein CNNs convert embedded protein sequences into fixed-size representations through stacked convolutions and max pooling.
- 2.3. Protein CNNs: Each protein token is converted into an embedding before entering the convolutional feature extractor.
- 2.3. Protein CNNs: Convolutional layers extract s-gram features across the protein sequence, with multiple layers stackable above the first layer.
- 2.3. Protein CNNs: Max pooling selects salient features from the final convolutional output to produce a fixed-size protein representation.The resulting vector is P_rep ∈ R^D_P.
2.4. Interaction Denses
The model concatenates molecule and protein representations, then uses interaction dense layers to estimate affinity scores for regression.
- Molecule and protein representations are concatenated into a joint input for the interaction dense layers.
- The interaction dense network approximates affinity scores with a multilayer feedforward network using dropout regularization.
- A final regression layer produces the output, and network weights are optimized using mean squared error.
3. Experiments
The experiments evaluate MT-DTI on Kiba and Davis using standardized datasets, pre-training data, model settings, and multiple performance metrics.
- MT-DTI is evaluated on the Kiba and Davis benchmarks, which are established datasets for drug-target interaction studies.
- Davis contains kinase-inhibitor dissociation constants, whereas Kiba combines Ki, Kd, and IC50 measurements into a consistency-optimized score.
- The experiments use five-fold cross-validation with a held-out test set and exclude compounds used during model training from the DrugBank case study.
- The model uses 97,092,853 canonical-SMILES molecules from PubChem for molecule-representation pre-training.
- The protein CNN uses an embedding layer, three convolutional layers, and max pooling, with dataset-specific filter sizes for Kiba and Davis.
- Evaluation uses MSE, concordance index, r^2_m, and AUPR, with binding labels defined by pKd ≥7 for Davis and KIBA score ≥12.1 for Kiba.
3.4. Baselines
MT-DTI outperforms the compared methods across the reported evaluation metrics, with larger improvements on Kiba and competitive performance even without fine-tuning.
- MT-DTI outperforms all compared methods on CI, MSE, r^2_m, and AUPR.
- MT-DTI tends to be more stable with larger training sets, achieving the lowest standard deviation for CI and AUPR.
- Without fine-tuning, MT-DTI produces competitive results, outperforming similarity-based methods and exceeding Deep-DTA on some metrics.
4. Case Study
A DrugBank case study ranks candidate compounds against EGFR, identifying established EGFR-targeting drugs and additional compounds not previously reported for EGFR association.
- The case study scores and ranks 1,794 selected DrugBank molecules against epidermal growth factor receptor.
- The model identifies known EGFR-targeting drugs, including Erlotinib, Gefitinib, Afatinib, Lapatinib, and Osimertinib, among high-affinity predictions.
- Lapatinib receives the highest predicted affinity among the examples discussed.
- The ranked predictions also include chemical compounds not previously reported as associated with EGFR, including drugs targeting opioid receptors, antihistamines, antipsychotics, and corticosteroids.
5. Related Work
Prior DTI research includes binary-classification and deep-learning approaches, alongside docking and similarity-based methods. Transfer learning is used as a strategy for improving task-specific results.
- DTI prediction has traditionally been framed as a binary classification problem, including recent interpretable deep-learning approaches.
- Existing methods span simulation-based molecular docking, similarity-matrix methods, and deep-learning models.Docking uses molecular and protein 3D features but is difficult to scale, while similarity-based methods address scalability through machine learning.
- The proposed model differs from prior deep-learning methods through improved molecule representation and transfer learning.Transfer learning uses pre-trained weights to improve results on corresponding tasks.
6. Discussion
The paper introduces a self-attention-based molecule representation, pre-trained on publicly available compound data and transferred to the MT-DTI model. MT-DTI is fine-tuned on two DTI benchmarks and outperforms existing methods across four evaluation metrics.
- The paper proposes a self-attention-based molecule representation for drug-target interaction prediction.The representation is pre-trained using publicly available compound data.
- Pre-trained representation parameters are transferred to MT-DTI, which is fine-tuned using two DTI benchmark datasets.
- MT-DTI outperforms all other existing methods across four evaluation metrics.
- The paper also includes a case study of identifying drug candidates targeting the cancer protein EGFR.