SSM-DTA: Breaking the Barriers of Data Scarcity in Drug-Target Affinity Prediction

Introduction

• Docking and molecular dynamics can predict DTA, but their high computational demands can make them costly.Source: Introduction · p. 2
• After filtering, BindingDB contains around 200k clean points from approximately 2 million binding data points, with noisy labels for some compound-protein pairs.The passage notes that this clean set is small relative to datasets often used in NLP and computer vision.Source: Introduction · p. 2
• SSM-DTA significantly improved DTA prediction accuracy on BindingDB, DAVIS, and KIBA and showed strong generalization to unknown drugs.Case studies also identified important atomic groups and amino acids within binding sites.Source: Introduction · p. 2
• The authors propose a semi-supervised multi-task training framework with strategies intended to alleviate data limitations in DTI prediction.They report state-of-the-art performance on benchmark datasets and studies of structural knowledge and binding information.Source: Introduction · p. 2

Drug-Target Interaction/Affinity Prediction

• Structure-based DTA methods use 3D drug and protein structures with force fields, while structure-free methods include similarity-based and deep-learning approaches.The passage names docking and molecular dynamics as structure-based examples, and KronRLS, SimBoost, and several neural models as structure-free examples.Source: Drug-Target Interaction/Affinity… · p. 2

Molecule and Protein Pre-training

• Molecule pretraining commonly uses SMILES Transformers or molecular-graph neural networks, while protein pretraining uses sequence Transformers; these approaches pretrain one modality at a time.Source: Molecule and Protein Pre-training · p. 2
• The SSM-DTA framework applies MLM to paired and unpaired data and uses cross-attention between paired drug-target sequences.Its two Transformer encoders take drug or molecule sequences and target or protein sequences as input.Source: Figure 1 · p. 3

Methods

• Paired drug SMILES and target FASTA sequences are Transformer-encoded; paired examples feed DTA regression and MLM, while unlabeled molecule and protein sequences feed MLM only.The paired drug and target sequences are also prepended with special [CLS] tokens.Source: Methods · p. 3
• The lightweight cross-attention uses each [CLS] representation to attend over the paired sequence; concatenated attended representations feed an MLP affinity head.Its cost is O((|D| + |T|) × d), versus O(|D| × |T| × d) for pairwise attention.Source: Methods · pp. 3–4, Semi-supervised Training with… · p. 4

Multi-task Training with MLM

• On labeled pairs, randomly masked drug and target tokens are reconstructed jointly with DTA prediction, maximizing the original paired data without adding affinity labels.The MLM objectives are jointly optimized with DTA prediction to enhance drug and target representations.Source: Multi-task Training with MLM · p. 4

Semi-supervised Training with Large-scale Unlabeled Molecule and Protein Data

• The semi-supervised objective combines labeled-pair regression with MLM on labeled and unlabeled molecules and proteins, unlike separate pretraining that misses drug–target interaction.The paper describes separate pretraining as inadequate for capturing the interaction critical to DTA prediction.Source: Semi-supervised Training with… · p. 4

Training Overview

• The final objective combines paired-data MSE with MLM losses for paired and unlabeled sequences, weighted by α and β, which are set equal.The unlabeled molecule and protein MLM losses are combined with paired drug–target MLM and MSE losses.Source: Training Overview · p. 4

Discussion

• Joint training combines unlabeled-data MLM and task-specific DTA representations, avoiding catastrophic forgetting reported for pretrain/fine-tune methods; it also increases training cost.The authors identify more efficient use of unlabeled data as future work.Source: Discussion · pp. 4–5

Experimental Settings

• BindingDB uses a 6:1:3 train/validation/test split, while DAVIS and KIBA use 7:1:2; unknown-drug evaluation is reported for DAVIS, not KIBA, because the latter setting was not open-sourced.BindingDB results use RMSE and PC, DAVIS and KIBA use MSE and CI, and unknown-drug DAVIS uses MSE and R2.Source: Experimental Settings · pp. 5–6

Performances on DAVIS and KIBA

• On KIBA, SSM-DTA improved the implemented baseline from 0.162 to 0.154 MSE and from 0.891 to 0.895 CI; the paper reports similar improvements on DAVIS.Table 2 reports mean (std) results across three runs.Source: Table 2 · p. 6, Performances on DAVIS and KIBA · p. 6

Performances on BindingDB

• On BindingDB IC50, SSM-DTA reduced RMSE from 0.787 to 0.712, a greater-than-7% improvement over the implemented baseline; on Ki, it achieved 0.792 RMSE and 0.863 PC.The paper also reports that it surpasses MONN (0.764) and BACPI (0.740) by about 3%–5%.Source: Performances on BindingDB · p. 6

Performances on DAVIS with Unknown Drug Setting

• In the DAVIS unknown-drug setting, SSM-DTA achieved 0.8019 MSE and 0.2803 R2, outperforming all baselines, including GeneralizedDTA.The authors describe this as superior generalization on DTA pairs with unknown drugs.Source: Performances on DAVIS with Unknown… · p. 6

Case Study of Drug-Target Binding

• For PCP–S100A4, 4 of the 10 highest-attention residues lie near the binding site; for flurazepam–GABAA, attention identifies four nearby amino acids.Complex structures were used to assess predictions, not for training; some high-attention residues can also lie outside the immediate binding-site vicinity.Source: Case Study of Drug-Target Binding · p. 7, Figure 2 · p. 8

Ablation Study

• Paired MLM lowers valid MSE from 0.625 to 0.545; unlabeled-data training lowers valid MSE from 0.625 to 0.601 and test RMSE from 0.787 to 0.772.With all components, valid/test results further improve to 0.513 and 0.712; the study reports each component positively affects prediction.Source: Ablation Study · p. 7, Table 4 · p. 7

Training Strategies Comparison

• SSM-DTA performs best among feature-based tuning, pre-training/fine-tuning, and semi-supervised multi-task training; feature-based tuning is limited by few tuned parameters.The authors argue separate pre-training overlooks drug-target interaction and favor SSM for paired interaction-related tasks.Source: Training Strategies Comparison · pp. 7–8, Table 5 · p. 8

Target Detection on DrugBank

• For 100 DrugBank drugs, SSM-DTA identifies 13 true targets at Top-1 versus 5 for baseline, and 18 at Top-5 versus 10.The experiment ranks each drug against 4,351 targets and suggests potential utility for drug repurposing.Source: Target Detection on DrugBank · p. 8

Virtual Screening on EGFR

• Among 6,848 DrugBank drugs ranked against EGFR, Afatinib ranked 8th (top 0.1%), illustrating potential virtual-screening utility.Source: Virtual Screening on EGFR · p. 8

Drug Feature Learning and Drug Grouping

• For 285 drugs across five DrugBank targets, SSM-DTA produces better-defined target-specific embedding clusters than the baseline, while DVMP shows only two apparent clusters.The figure colors drugs by target and compares embeddings from DVMP, the baseline, and SSM-DTA.Source: Drug Feature Learning and Drug Grouping · p. 9, Figure 3 · p. 9

Training Analysis

• SSM-DTA converges faster and has lower training and validation MSE than the baseline, with similar converged training MSE but substantially better validation MSE.The curves show lower MSE across training on both sets, with the clearest converged difference on validation.Source: Training Analysis · p. 9, Figure 4 · p. 9

Effects of Unlabeled Data and MLM Loss Weight

• Validation MSE improves progressively as unlabeled molecule and protein data scale increases from 1M to 10M; larger scales were not tested due to resource limits.The authors suggest greater data diversity may improve generalized representations.Source: Effects of Unlabeled Data and MLM Loss… · p. 9, Figure 5 · p. 10
• Validation MSE is lowest at shared MLM-loss weight 5 among {0, 1, 5, 10, 20}, in experiments without cross-attention.The authors describe a trade-off because the MLM objective differs from DTA prediction.Source: Effects of Unlabeled Data and MLM Loss… · p. 9, Figure 5 · p. 10

Effectiveness of our Cross-attention Module

• SSM-DTA's cross-attention outperforms DeepAffinity's pairwise attention on DAVIS and BindingDB Ki and is more computationally efficient.Table 6 reports the comparison; the passage also notes lower computation cost.Source: Effectiveness of our Cross-attention… · p. 10, Table 6 · p. 10

Conclusion

• SSM-DTA combines paired MLM training, semi-supervised learning with unlabeled molecules and proteins, and an efficient cross-attention module for drug-target interaction.The conclusion reports improved performance on multiple DTA benchmarks, generalization to unknown drugs, and identification of key atom groups and amino acids in three case studies.Source: Conclusion · p. 10
• Grouping drugs by target revealed target-specific features embedded in SSM-DTA, which the authors offer as an explanation for its improved performance.Source: Conclusion · p. 10
• The authors propose exploring an efficient unification of structure- and sequence-based methods in future work.Source: Conclusion · p. 10

Limitations

• BindingDB IC50 training (263,583 training samples) took 8 NVIDIA V100 GPUs for 4.8 days, making the joint labeled and unlabeled training process relatively costly.Source: Limitations · p. 10
• Benchmark performance has not yet been experimentally validated, leaving a risk of dataset-specific overfitting and uncertain real-world affinity prediction.Source: Limitations · p. 10
• The authors identify generalization across datasets as a broader challenge, given variation in drugs, targets, and affinity types.Source: Limitations · p. 10

Experimental Settings

• The experiments evaluate DTA prediction against deep-learning baselines on the stated benchmark datasets using the described experimental settings.Source: Experimental Settings · p. 13

Datasets

• BindingDB, DAVIS, and KIBA use random train/validation/test splits; BindingDB includes IC50 and Ki, DAVIS uses Kd, and KIBA combines Ki, Kd, and IC50 bioactivities.BindingDB IC50 has 376,751 samples; DAVIS has 30,056 pairs; KIBA has 118,254 pairs.Source: Datasets · p. 13

Model Configurations

• The model uses two 12-layer RoBERTa-base Transformer encoders, 768-dimensional embeddings, and a two-layer regression head, totaling 178M parameters.The reported inference speed is 0.0061 seconds per sample.Source: Model Configurations · p. 13

Training and Evaluation

• Training uses Adam with learning rate 1e−4, coefficient weights α and β set to 2.0, and evaluation metrics selected by dataset.BindingDB is evaluated with RMSE and Pearson correlation; DAVIS and KIBA use MSE and concordance index.Source: Training and Evaluation · p. 13

Compared Baselines

• DeepDTA and DeepAffinity were evaluated on BindingDB, while MONN and BACPI were also compared on that dataset.DeepDTA was evaluated on BindingDB by MONN; DeepAffinity, MONN, and BACPI are described as BindingDB baselines.Source: Compared Baselines · pp. 13–14
• Compared baselines include KronRLS, GraphDTA, DeepPurpose, SAGDTA, and MGraphDTA, which were evaluated on DAVIS and KIBA.Source: Compared Baselines · pp. 13–14

• Protein inputs contain a [cls]T token and amino-acid tokens, while drug inputs use SMILES and the same encoding process.Source: Encoding Process of Transformer Encoder · p. 14
• Each amino acid is embedded, then receives a position-specific encoding to form a position-aware input vector.Source: Encoding Process of Transformer Encoder · p. 14
• Multi-head attention computes scaled query-key dot-product scores followed by softmax, then concatenates and linearly transforms the head outputs.Source: Encoding Process of Transformer Encoder · p. 14
• Repeated attention and feed-forward layers produce a protein representation containing the whole-sequence [cls]T vector and amino-acid hidden states.Source: Encoding Process of Transformer Encoder · p. 14