Beyond Random Splits: Evaluating Drug-Target Affinity Models Under Chemically and Biologically Motivated Distribution Shifts Copy

1 Introduction

• Chemical cold-start requires transferring structure–activity relationships, whereas target cold-start requires transferring interaction rules across sequence space.These are distinct extrapolation tasks, unlike warm-start evaluation where both entities appear during training.Source: §1 · p. 1
• The benchmark evaluates the same architecture family under four generalization regimes to examine absolute error and model-selection stability separately.Source: Figure 1 · p. 2
• Random pair-level splits can place similar compounds or homologous proteins on both sides of the split, yielding optimistic estimates of generalization.The paper also notes that canonical Davis and KIBA benchmarks have limited chemical and target diversity.Source: §1 · p. 2
• Prior architecture studies typically use chemical- or target-held-out settings to assess a proposed model, rather than test whether architecture rankings remain stable across regimes.Source: §1 · p. 2
• The study compares a Morgan-fingerprint + protein-CNN baseline with 12 architectures spanning four drug representations and three ESM-2 interaction modes.The controlled design evaluates these models under the same distribution-shift regimes.Source: §1 · p. 2

2.1 Dataset

• 718,800 unique drug-protein pairs span 273,528 compounds and 3,704 proteins after curation and measurement aggregation.Ki measurements from ChEMBL and BindingDB were converted to pKi; repeated measurements for each pair were mean-aggregated before splitting.Source: §2.1 · pp. 2–3

2.2 Evaluation regimes

• The four OOD regimes hold out scaffold groups, fingerprint-similarity clusters, protein-sequence clusters, or both chemical and protein groups.For dual OOD, an interaction is retained only when the independently assigned drug and protein partitions agree, so neither side is represented in training.Source: §2.2 · p. 3

2.3 Architecture matrix and training

• A 4 × 3 design combines ECFP4, ChemBERTa, GCN, or fused GCN+ChemBERTa with ESM-2 mean pooling, attention pooling, or drug-conditioned cross-attention.A Morgan-fingerprint + character-level protein-CNN baseline provides a non-pretrained protein-encoder comparison.Source: §2.3 · p. 3
• Models are trained independently within each regime and ranked by RMSE, with Kendall’s τ as the primary measure of ranking agreement and Spearman’s ρ as a secondary check.The analysis also includes split-severity statistics and sample-size controls.Source: §2.3 · p. 3

3.1 Distribution shift dominates absolute error

• Mean validation RMSE rises from 0.950 under scaffold OOD and 0.945 under fingerprint OOD to 1.299 under protein OOD and 1.321 under dual OOD.The increases relative to scaffold OOD are 36.7% for protein OOD and 39.0% for dual OOD.Source: §3.1 · pp. 3–4

3.2 Architecture rankings depend on the regime

• ECFP+AttnPool ranks first under scaffold, fingerprint, and protein OOD but last under dual OOD, while GCN+CrossAttn ranks first under dual OOD.Their relative advantage shifts from 0.104 RMSE in favor of ECFP+AttnPool under scaffold OOD to 0.092 in favor of GCN+CrossAttn under dual OOD.Source: Figure 2 · p. 4, §3.2 · p. 4

3.1 Distribution shift dominates absolute error

• Held-out evaluation preserves the error-severity pattern, with mean RMSE of 0.986, 0.946, 1.354, and 1.368 for scaffold, fingerprint, protein, and dual OOD, respectively.The protein-OOD winner changes from ECFP+AttnPool on validation to GCN+ChemBERTa+CrossAttn on the held-out test set.Source: §3.1 · p. 4

3.2 Architecture rankings depend on the regime

• Scaffold and fingerprint rankings agree (τ = 0.79), but agreement falls for scaffold versus protein OOD (τ = 0.39) and reverses for scaffold versus dual OOD (τ = −0.55).Fingerprint versus dual OOD has the largest inversion (τ = −0.64); the top three scaffold and dual OOD architectures do not overlap.Source: §3.2 · p. 4

4 Discussion and conclusion

• The baseline also shows regime dependence, ranking lowest on chemical-OOD validation splits, highest under protein OOD, and lowest again under dual OOD.This extends the ranking changes beyond the controlled ESM-2 architecture matrix.Source: §4 · p. 5
• Repeated seeds, size-matched protein-OOD subsets, and group-aware bootstrap intervals support an architecture-specific response to dual OOD rather than a simple training-size effect.The pattern is consistent with different architectural inductive biases becoming useful under different extrapolation forms.Source: §4 · p. 5
• Similarity-aware model selection is proposed as a potential mitigation, using chemical and target similarity to route predictions among architectures.The benchmark distinguishes dual OOD, with median Tanimoto 0.619 and Jaccard 0.142, from protein OOD, with median Tanimoto 1.000 and Jaccard 0.139.Source: §4 · p. 5
• The authors recommend evaluating characterized chemical and target shifts and reporting ranking stability, since a single aggregate benchmark number can obscure deployment-relevant model preferences.Source: §4 · p. 5
• The study is retrospective and sequence-based; heterogeneous public assay contexts and proxy split definitions bound how directly the results represent prospective discovery.The authors plan to test whether ranking behavior extends to larger pretrained encoders and alternative graph architectures.Source: §4 · p. 5

A.1 Split-severity audit

• Protein-focused regimes have median nearest-training 3-mer Jaccard values of 0.139 and 0.142, and about 32% of their test proteins fall below 0.10.Protein OOD retains familiar compounds (median Tanimoto 1.000), whereas dual OOD also shifts chemical space (0.619).Source: §A.1 · p. 7
• The pKi distribution figure may mark pKi = 5, corresponding to Ki = 10 µM, as a reference rather than a universal activity threshold.Source: Figure 3 · p. 7

A.2 Training and analysis details

• Training uses AdamW, cosine annealing, a maximum of 40 epochs, and early stopping; frozen ESM-2 and ChemBERTa encoders leave task-specific modules trainable.Trainable parameter counts range from 0.62M to 2.09M.Source: §A.2 · p. 8

A.3 Rank-correlation analysis

• Spearman ρ is 0.895 between scaffold and fingerprint OOD rankings, but −0.727 between scaffold and dual OOD rankings.The table reports correlations between architecture rankings across evaluation regimes; negative values indicate inverted orderings.Source: §A.3 · p. 8, Table 1 · p. 8

B.1 Additional CNN baseline

• The Morgan-fingerprint + protein-CNN baseline ranks below all 12 core architectures under chemical OOD, first under protein OOD, and last under dual OOD.Its validation RMSE is 1.0355/1.0968 under scaffold/fingerprint OOD, 1.2505 under protein OOD, and 1.4700 under dual OOD.Source: §B.1 · p. 8, Table 2 · p. 8

• ECFP+AttnPool has RMSE 0.8810 ± 0.0022, 1.2544 ± 0.0145, and 1.3652 ± 0.0358 under scaffold, protein, and dual OOD, respectively.Source: §B.2 · p. 8
• GCN+CrossAttn has RMSE 0.9947 ± 0.0076, 1.3297 ± 0.0272, and 1.3203 ± 0.0385 under scaffold, protein, and dual OOD, respectively.Source: §B.2 · p. 8
• ChemBERTa+AttnPool provides an intermediate reference, with RMSE 0.9305 ± 0.0041, 1.3053 ± 0.0105, and 1.3242 ± 0.0092 under scaffold, protein, and dual OOD, respectively.Source: §B.2 · p. 8

B.3 Training-size ablation

• At 352k training pairs, GCN+CrossAttn outperforms ECFP+AttnPool by 0.0924 RMSE under dual OOD; reduced training size alone does not reproduce the reversal.Across five protein-OOD downsampling draws, ECFP+AttnPool and GCN+CrossAttn have RMSEs of 1.2804 ± 0.0246 and 1.2917 ± 0.0315, respectively.Source: §B.3 · p. 9, Table 3 · p. 9

B.4 Group-aware bootstrap uncertainty

• The bootstrap 95% CI favors ECFP under scaffold OOD (−0.1113, [−0.1275, −0.0956]) and fingerprint OOD (−0.0860, [−0.1014, −0.0710]).Negative RMSE differences favor ECFP; the resampling was paired at the corresponding split-group level.Source: §B.4 · p. 9
• Under dual OOD, the bootstrap difference reverses and favors GCN+CrossAttn (+0.2059, [0.0153, 0.3851]), while the protein-OOD interval includes zero (−0.0555, [−0.2306, 0.0968]).The intervals are 95% CIs from 10,000 paired group-level bootstrap resamples.Source: §B.4 · p. 9

C.1 Model configuration

• ESM-2 (480 dimensions) and ChemBERTa (384 dimensions) are frozen during training, focusing comparison on drug representation and interaction design rather than end-to-end fine-tuning.Source: §C.1 · p. 9
• Table 4 reports architecture-specific batch sizes and trainable parameter counts for task-specific modules, excluding the frozen pretrained representations.Source: Table 4 · p. 9

• Table 5 reports validation RMSE across the architecture-by-split matrix.Source: Table 5 · p. 10
• Lower RMSE is better, and bold marks the best model within each regime.Source: Table 5 · p. 10
• The final row gives the column mean across all 12 architectures.Source: Table 5 · p. 10

• Table 6 reports held-out test RMSE across the architecture-by-split matrix.Source: Table 6 · p. 10
• Lower RMSE is better, and bold marks the best model within each regime.Source: Table 6 · p. 10
• The final row gives the column mean across all 12 architectures.Source: Table 6 · p. 10