PaperlayerView exact revision on arXiv

Source-linked AI summary

Drug-Target Interaction Prediction via Hierarchical Sequential Cross-Attention over Chemical and Protein Language Models

Khadidja Henni, Hamza Abdelali, Abdelkrim Aries, Neila Mezghani, Brigitte Vannier, Sara Magdouli, Lina Abou-Abbas

arXiv:2609.34921v1cs.LGcs.AI

TL;DR

Many sequence-based DTI models combine drug and protein representations only at the final prediction stage, leaving their cross-molecular dependencies insufficiently modeled. This paper proposes a sequence-only architecture that uses pre-trained sequence encoders, CNN condensation, sequential bidirectional cross-attention, and attention-based pooling. It reports strong and competitive results across three datasets, with 25.2 million trainable parameters, and encouraging generalization to unseen protein targets.

  • Problem

    Many sequence-based DTI models encode drugs and proteins independently and fuse them late, limiting explicit modeling of dependencies between drug substructures and protein regions.

    Source: §I · p. 1, C. Attention-Based Interaction Modeling · p. 2

  • Method

    The model combines ChemBERTa and ESM-2 sequence encoders with CNN feature extraction, sequential bidirectional cross-attention, and attention-based pooling.

    Source: §I · pp. 1–2

  • Results

    The method reports strong and competitive results on BIOSNAP, Davis, and BindingDB with 25.2 million trainable parameters; it performs best on BIOSNAP.

    Source: F. Discussion and Limitations · p. 5, §VI · p. 6

  • Takeaways & Limitations

    Cold-start evaluation indicates encouraging generalization, particularly to unseen protein targets.

    Source: §VI · p. 6

    The sequence-only method omits molecular graphs, 3D protein structures, docking poses, and binding-site geometry, which may limit performance when structural information is decisive.

    Source: F. Discussion and Limitations · p. 6

Abstract

from arXiv · show

Predicting Drug-Target Interactions~(DTIs) is a central task in computational drug discovery, with direct applications in virtual screening, drug repurposing, and therapeutic candidate prioritization. Although recent deep learning methods have improved DTI prediction, many sequence-based models still process drugs and proteins independently and only combine their representations at a late prediction stage. This limits their ability to explicitly model cross-molecular dependencies between chemical substructures and protein sequence regions. In this paper, we propose a sequence-only DTI prediction architecture that combines two pre-trained language models, ChemBERTa for drug SMILES strings and ESM-2 for protein amino acid sequences, with a hierarchical interaction module. The proposed model first extracts contextual representations using pre-trained encoders, then applies 1D convolutional layers to condense local sequence patterns, followed by a sequential bidirectional cross-attention mechanism inspired by the induced-fit view of molecular recognition. Finally, attention-based pooling constructs fixed-size interaction-aware vectors for binary prediction. Experiments on BIOSNAP, Davis, and BindingDB show that the proposed model achieves the best performance on BIOSNAP, matches the best AUROC on Davis, and remains competitive on BindingDB while using only 25.2 million trainable parameters. Ablation results confirm the contribution of both the CNN and cross-attention modules, and cold-start experiments indicate promising generalization to unseen proteins and drugs.

I. INTRODUCTION

DTI prediction can help prioritize candidates across drug discovery, but many sequence-based models fuse drug and protein representations only at the final stage. The paper proposes a sequence-only architecture that models their interaction hierarchically and evaluates it through ablation, cold-start, and cross-dataset experiments.

I. INTRODUCTION

  • DTI prediction supports virtual screening, drug repurposing, mechanism-of-action analysis, and early safety assessment in a costly, high-risk drug-development process.The passage notes that drug discovery can require more than a decade of research and substantial financial investment.Source: §I · p. 1
  • Late fusion may miss fine-grained dependencies between drug substructures and protein regions, while global pooling can discard position-specific information.The introduction frames these as two limitations of many sequence-based models.Source: §I · p. 1
  • The architecture combines ChemBERTa and ESM-2 with CNN-based local feature extraction, sequential bidirectional cross-attention, and attention-based pooling.The sequential module first updates protein representations using drug context, then updates drug representations using the refined protein context.Source: §I · pp. 1–2
  • The proposed sequential cross-attention improves over its parallel counterpart on AUROC, AUPRC, and sensitivity, with comparable specificity.The design is inspired by the induced-fit view of molecular recognition, without explicitly simulating 3D conformational changes.Source: §I · p. 2
  • Ablation, cold-start, and cross-dataset experiments report measurable contributions from the architectural components and competitive results with 25.2M trainable parameters.The architecture is sequence-only and does not use molecular graphs or structural data.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

The background reviews sequence representations for drugs and proteins and traces DTI prediction from conventional deep learning to Transformer and hybrid approaches. It also notes that richer graph-based or hybrid approaches can bring higher architectural complexity and larger parameter counts.

A. Drug and Protein Representations

  • SMILES provides a compact textual encoding of molecular structure suited to sequence models, while protein amino acid sequences can be processed by language models trained on large biological databases.The cited background notes that SMILES does not explicitly describe 3D conformations and that protein language models can capture contextual and evolutionary signals.Source: A. Drug and Protein Representations · p. 2

B. Deep Learning for DTI Prediction

  • DTI models range from CNNs, recurrent networks, and graph neural networks to Transformer and hybrid methods, with some newer hybrids trading higher accuracy for greater complexity and parameter counts.Examples include DeepDTA, WideDTA [11], DeepAffinity, GraphDTA, and MolTrans.Source: B. Deep Learning for DTI Prediction · p. 2

C. Attention-Based Interaction Modeling

The section motivates sequential cross-attention as a response to interaction models that do not exploit dependencies between refinement steps. It formulates DTI prediction as binary classification and outlines the architecture’s sequence-encoding, fusion, pooling, and prediction stages.

C. Attention-Based Interaction Modeling

  • Many cross-attention approaches use a single step or simultaneous directions from original representations, motivating a sequential hierarchical interaction module.The broader discussion describes cross-attention as allowing one modality to query another for relevant regions.Source: C. Attention-Based Interaction Modeling · p. 2

A. Problem Formulation

  • Given a drug SMILES string and protein amino acid sequence, the task is to predict a binary interaction label using learned encoder representations.The labels are y=1 for an interaction and y=0 for no known interaction.Source: A. Problem Formulation · p. 2

B. Overall Architecture

  • The architecture encodes sequences, condenses features with CNNs, applies sequential bidirectional cross-attention, pools attention-weighted features, and predicts with an MLP.Figure 1 presents the proposed architecture’s stages.Source: B. Overall Architecture · p. 2, Figure 1 · p. 3

C. Pre-trained Sequence Encoders

The method uses pre-trained language models for drug and protein sequences, partially fine-tunes them, and condenses their contextual embeddings with CNNs. Sequential cross-attention then refines the two representations in order, using the refined protein context in the drug-update step.

C. Pre-trained Sequence Encoders

  • ChemBERTa encodes drug SMILES and ESM-2 encodes protein sequences; only the top two Transformer layers of each are unfrozen during fine-tuning.Partial fine-tuning reduces trainable parameters from approximately 193 million to 25.2 million.Source: C. Pre-trained Sequence Encoders · p. 3

D. CNN-Based Local Feature Extraction

  • Independent 1D CNN blocks capture local chemical fragments or protein motifs and halve sequence length to make subsequent cross-attention more efficient.Each block uses a kernel-size-3 convolution with 128 output channels, ReLU, and kernel-size-2 max pooling.Source: D. CNN-Based Local Feature Extraction · p. 3

E. Sequential Bidirectional Cross-Attention

  • Sequential cross-attention first refines protein features using drug context, then uses the refined protein representation to update drug features.This differs from a parallel design that uses the original representations independently and is inspired by induced fit without claiming to simulate 3D changes.Source: E. Sequential Bidirectional… · p. 3

F. Attention-Based Pooling and Prediction

Learned attention scores weight sequence positions into a fixed-size representation, emphasizing pharmacologically relevant positions. The pooled drug and protein vectors are then combined for binary prediction.

F. Attention-Based Pooling and Prediction

  • Learnable position scores are normalized to weight hidden states into a fixed-size representation focused on pharmacologically relevant positions.The method replaces standard global pooling, which discards position-level information.Source: F. Attention-Based Pooling and… · p. 4
  • The pooled drug and protein vectors are concatenated and passed through a three-layer MLP, whose sigmoid output gives the interaction probability.The hidden layers use ReLU and dropout with p = 0.2.Source: F. Attention-Based Pooling and… · p. 4

A. Datasets

The evaluation covers three DTI datasets with different label characteristics and uses pair-level and cold-start test settings. For Davis and BindingDB, training is balanced while validation and test retain their original distributions.

A. Datasets

  • Davis contains kinase-inhibitor and kinase-protein binding measurements, binarized using Kd < 30 nM; this yields a severely imbalanced dataset with an approximately 1:6 ratio.BindingDB binary labels use Kd < 1 µM. BIOSNAP is balanced, with validated positive interactions and sampled negative pairs in equal proportion.Source: A. Datasets · p. 4
  • For Davis and BindingDB, training sets are resampled to balance classes, while validation and test retain their original distributions without added negative sampling.Table I reports interaction-pair counts as train/validation/test.Source: A. Datasets · p. 4, Table I · p. 4

B. Evaluation Protocol

  • Pair-level splits test new interactions among known entities, whereas cold-start splits hold out 20% of drugs or proteins and test their interactions.Cold-start evaluates generalization to entities not observed during training.Source: B. Evaluation Protocol · p. 4
  • Proposed-model experiments use five independent runs, but baseline comparisons are point estimates from potentially nonidentical splits, so small differences warrant caution.A fully controlled comparison reporting mean and standard deviation across all seeds is left for future work.Source: B. Evaluation Protocol · p. 4

C. Training Configuration

Across the three benchmarks, the model leads on BIOSNAP AUROC and AUPRC, matches the best Davis AUROC, and remains competitive on BindingDB. Results are compared with published or otherwise reported baseline values, and performance is assessed with four complementary metrics.

C. Training Configuration

  • The model is implemented in PyTorch and PyTorch Lightning, with early stopping based on validation AUROC after three epochs without improvement.Table II reports the main hyperparameters.Source: C. Training Configuration · p. 4, Table II · p. 4

D. Metrics

  • AUROC assesses separation across thresholds, AUPRC focuses on positive-class performance, sensitivity measures detected interactions, and specificity measures correctly rejected non-interactions.AUPRC is especially relevant for imbalanced datasets.Source: D. Metrics · p. 5

A. Comparison with State-of-the-Art Methods

  • The comparison spans CNN, Transformer, PLM fine-tuning, dual-PLM, cross-attention, and multi-scale fusion methods, using published or comparable reported baseline values.It is a literature-level evaluation, not a fully controlled reimplementation of all baselines.Source: A. Comparison with State-of-the-Art… · p. 5
  • BIOSNAP leads AUROC (0.920) and AUPRC (0.924); Davis matches best AUROC (0.920) but has lowest AUPRC (0.356); BindingDB is competitive at AUROC (0.920) and AUPRC (0.642).The cited comparison table covers all three datasets.Source: A. Comparison with State-of-the-Art… · p. 5
  • On BIOSNAP, sensitivity (0.854) and specificity (0.830) are slightly below Fine-tuning BERT, despite the model leading overall.On Davis, sensitivity is highest at 0.901; on BindingDB, HyperAttentionDTI has the best AUPRC (0.876) and DLM-DTI the best specificity (0.916).Source: A. Comparison with State-of-the-Art… · p. 5

B. Integrated Dataset Evaluation

Integrated training improves AUPRC on Davis and BindingDB but lowers it on BIOSNAP; cold-start results and ablations also support the model’s generalization and component contributions.

B. Integrated Dataset Evaluation

  • Davis AUPRC rose +0.092 (0.356→0.448) and BindingDB rose +0.122 (0.642→0.764), while BIOSNAP fell from 0.924 to 0.840 under integrated training.The authors suggest that mixing datasets with different distributions and label-generation protocols may explain the BIOSNAP decrease.Source: B. Integrated Dataset Evaluation · p. 5

C. Cold-Start Generalization

  • 0.87 ROC-AUC on unseen proteins outperformed MolTrans (0.77) and DeepDTA (0.85), while all three models reached 0.85 on unseen drugs.The evaluation held out 20% of drugs or proteins independently.Source: C. Cold-Start Generalization · p. 5

D. Ablation Study

  • Removing cross-attention lowered AUROC from 0.920 to 0.911, removing CNN lowered it to 0.916, and removing both lowered it to 0.905.The authors identify cross-attention as the most important post-encoder component and describe the CNN and cross-attention contributions as complementary.Source: D. Ablation Study · p. 5
  • AUROC was 0.920 with sequential attention versus 0.914 with parallel attention; sequential attention also led on AUPRC and sensitivity, while parallel attention slightly led on specificity.The respective pairs were AUPRC 0.924 vs. 0.920, sensitivity 0.854 vs. 0.833, and specificity 0.830 vs. 0.834.Source: D. Ablation Study · p. 5

E. Model Complexity

The architecture uses relatively few trainable parameters while remaining competitive, but results and stated limitations identify boundaries for sequence-only prediction and benchmark comparisons.

E. Model Complexity

  • 25.2 million trainable parameters are used, representing reductions of 2.5× versus MolTrans, 3.4× versus DLM-DTI, and 14× versus full BERT fine-tuning.The approximately 193-million-parameter model fine-tunes only the top two layers of each encoder and the newly added modules.Source: E. Model Complexity · p. 5

F. Discussion and Limitations

  • BIOSNAP is the strongest setting, while Davis precision and BindingDB AUPRC expose limits under class imbalance and chemical diversity.The BindingDB AUPRC gap relative to HyperAttentionDTI, which uses structural information, is described as pointing to a ceiling for purely sequence-based approaches on chemically complex data.Source: F. Discussion and Limitations · pp. 5–6
  • The sequence-only method omits molecular graphs, 3D structures, docking poses, and binding-site geometry; attention weights also need validation before biological interpretation.The authors also caution that some baseline values may use different preprocessing and split protocols, limiting strict comparisons without controlled reimplementation.Source: F. Discussion and Limitations · p. 6

VI. CONCLUSION

The conclusion presents a hierarchical sequence-based DTI model with competitive results and encouraging cold-start generalization, while outlining future work on representations, validation, and benchmarking.

VI. CONCLUSION

  • The model combines ChemBERTa and ESM-2 with CNN feature extraction, sequential bidirectional cross-attention, and attention-based pooling, achieving strong and competitive results with 25.2 million trainable parameters.Ablations found measurable improvements from CNN condensation and sequential cross-attention; cold-start evaluation showed encouraging generalization, particularly to unseen proteins.Source: §VI · p. 6
  • Future work proposes richer chemical or graph representations, protein structural or binding-site features, validation of attention maps, and controlled benchmarks with reimplemented baselines and multiple seeds.The proposed additions are intended to address AUPRC gaps, provide biological grounding, support interpretability claims, and strengthen empirical comparisons.Source: §VI · pp. 6–7

GENERATIVE AI USAGE STATEMENT

  • Generative AI tools were used only for language polishing and formatting, and the authors state that they reviewed and verified the revisions.The authors attribute all scientific contributions, experiments, results, and conclusions to themselves.Source: GENERATIVE AI USAGE STATEMENT · p. 7

Related papers

Loading 2609.34921v1…