Source-linked AI summary
Benchmarking Peptide-Protein Affinity Prediction Across Peptide and Target Shifts
Jiaxin Tian, Darren An, Jun Li
TL;DR
Existing evaluations often use a single split, leaving unclear whether peptide–protein affinity models interpolate among observed entities or generalize across peptide and target shifts. This study integrates and deduplicates binding data and benchmarks representations, learners, and design choices across three partitions. Performance and model choice vary with the held-out entity, while representation differences narrow with more supervision and tested adaptation or simple fusion adds no consistent gain.
Problem
Single-split evaluations leave unclear whether peptide–protein affinity models interpolate among observed targets or generalize across peptide and target shifts.
Method
The benchmark integrates three quantitative binding-data sources and evaluates ten peptide representations with ESM-2 protein embeddings and six regressors across peptide-similarity, within-target, and leave-target-out partitions.
Results
Performance and selected model configurations changed with the held-out entity: mean SPCC was 0.462 for peptide-similarity, 0.669 within-target, and 0.530 for leave-target-out evaluation.
Takeaways & Limitations
Representation superiority is conditional on the intended generalization regime, so benchmarks should align partitions with use and jointly compare data scale, representations, and learners.
Takeaways & Limitations
The dataset pools Kd, Ki, and IC50 measurements on one transformed scale, while leave-target-out excludes identical sequences but not homologous proteins.
Abstract
from arXiv · showhide
Peptide-protein affinity models are often evaluated with a single data split, obscuring whether they interpolate among measurements for observed targets or generalize across peptide or target shifts. We integrated three sources of quantitative peptide-protein binding data to obtain 11,349 deduplicated pairs and benchmarked ten peptide representations, ESM-2 protein embeddings, and six regressors under peptide-similarity, within-target, and leave-target-out partitions. Across 60 matched representation-regressor configurations, mean test Spearman correlations were 0.462, 0.669, and 0.530, respectively. The top configuration shifted from ECFP-16 count fingerprints with random forest in the first two settings to HELM-BERT with Extra Trees when exact target sequences were excluded. Representation-rank correlations ranged from -0.042 to 0.624 across partitions, whereas regressor-rank correlations ranged from 0.771 to 0.943. Learning curves showed that representation differences were largest with limited supervision and narrowed as training data increased. PeptideCLM-2 adaptation and simple element-wise interaction features provided no consistent gain over a frozen encoder and direct concatenation under the tested protocols. These conclusions are specific to a dataset that pools transformed Kd, Ki, and IC50 measurements and to target exclusion at the exact-sequence level. Peptide-protein affinity benchmarks should therefore align data partitions with the intended use and jointly assess the effects of data scale, molecular representation, and downstream learner.
1. Introduction
Peptide–protein affinity prediction requires evaluation across distinct generalization regimes because random splits can favor interpolation and obscure transfer to novel peptides or targets. This benchmark compares peptide and protein representations, learners, data scales, encoder adaptation, and feature fusion across intended uses.
- Motivation: Peptides combine high affinity and selectivity with chemical accessibility, but proteolysis, short exposure, poor absorption, and low permeability constrain development.Cyclization, backbone modification, terminal capping, and non-canonical residues can mitigate these liabilities while enlarging the design space.
- Motivation: Computational models can prioritize peptide–target pairs using sequence, graph, fingerprint, or pretrained peptide and protein representations.Earlier models included physicochemical sequence kernels, while newer systems address motif–domain binding, interaction classification, and residue-level binding sites.
- Benchmark rationale: Random record-level splits may place close analogues, homologous sequences, or repeated biological entities in both training and test sets, favoring interpolation and overstating prospective performance.Representation rankings also depend on peptide class and downstream learner, so representation quality cannot be considered independently of partition and learner.
- Benchmark rationale: Three deployment regimes test structurally separated peptides, follow-up measurements for observed targets, and exact protein sequences excluded from fitting.The first two settings can exploit target-specific information from other records, whereas leave-target-out requires transfer through the protein representation and peptide–protein mapping.
- Study objective: The benchmark integrates and deduplicates three quantitative binding sources, then compares ten peptide representations, ESM-2 protein embeddings, six regressors, learning curves, encoder adaptation, and feature fusion.Its objective is interpretable conclusions across intended uses rather than a universal winning model.
2. Materials and Methods
The benchmark constructs a deduplicated peptide–protein affinity dataset, represents peptides and proteins independently, and evaluates matched representation–regressor configurations under multiple generalization-oriented partitions. It also measures data-scale effects, encoder adaptation, interaction-aware fusion, and statistical consistency using SPCC.
- Data sources: 11,349 deduplicated peptide–target pairs comprise the final benchmark, covering 9,510 peptides and 2,659 protein sequences from three quantitative binding-data sources.Records were keyed by peptide SMILES and complete target sequence before duplicate pairs were removed.
- Representations: Ten peptide featurizations span binary and count ECFP-16 fingerprints, chemical and peptide language models, and graph-based PepLand, paired with fixed-length ESM-2 protein features.In the primary benchmark, pretrained neural encoders were frozen to produce fixed-length peptide features.
- Pair construction: Peptide and target vectors were separately projected before direct concatenation or interaction-aware fusion with element-wise products and absolute differences.Both fusion rules used matched partitions, seeds, projection dimensions, and MLP prediction heads.
- Generalization-oriented partitions: Peptide-similarity, within-target, and leave-target-out partitions respectively separate peptides, retain observed targets across subsets, or group each exact protein sequence into one subset.Leave-target-out prevents exact-sequence overlap but imposes no homology or protein-family threshold.
- Regression and tuning: Six regressors cover linear, kernel, tree, boosted-tree, and neural approaches, with Optuna selecting configurations using training-only five-fold cross-validation SPCC.Selected models were refitted on the full training subset and evaluated on fixed test subsets across five random seeds.
- Scale and adaptation: Learning curves evaluate five peptide-specific representations with all six regressors at training sizes from 100 to 8,172 records under a fixed peptide-similarity partition.Encoder adaptation compares frozen, partially unfrozen, fully fine-tuned, and LoRA strategies using fixed protein features and concatenation.
- Evaluation and statistics: SPCC is the primary metric, and scenario comparisons use seed-averaged test SPCC for each of 60 matched representation–regressor configurations.Wilcoxon tests with Holm adjustment assess consistency across configurations, while AULC summarizes complete adaptation learning curves.
3. Results
Performance varied substantially across peptide-similarity, within-target, and leave-target-out evaluations, with both representation rankings and the best configuration changing by partition. More supervision reduced representation differences, whereas learner differences and training-set effects remained important; encoder adaptation and interaction-aware fusion added no consistent benefit.
- 3.1. The held-out entity changes both performance and model choice: 0.462, 0.669, and 0.530 mean test SPCC were obtained for peptide-similarity, within-target, and leave-target-out evaluation, respectively.The corresponding median SPCC values were 0.466, 0.707, and 0.524.
- 3.1. The held-out entity changes both performance and model choice: ECFP-16 count fingerprints with random forest led peptide-similarity and within-target evaluation, while HELM-BERT with Extra Trees led leave-target-out evaluation.The respective top scores were 0.545, 0.771, and 0.605.
- 3.1. The held-out entity changes both performance and model choice: Representation-rank correlations ranged from -0.042 to 0.624 across partition comparisons, whereas regressor-rank correlations ranged from 0.771 to 0.943.Partition choice therefore affected representation rankings more strongly than regressor rankings.
- 3.2. Additional supervision narrows representation differences but not learner differences: Extra Trees, random forest, and LightGBM reached full-data mean SPCC values of 0.526, 0.518, and 0.513 under peptide-similarity evaluation.Their 100-record values were 0.186, 0.170, and 0.150, and all three continued improving beyond 1,000 records.
- 3.2. Additional supervision narrows representation differences but not learner differences: HELM-BERT had the highest representation mean at every displayed training size, while the best–worst representation gap narrowed from 0.111 at 100 records to 0.033 with full data.Representation choice mattered most with limited supervision, whereas downstream learner differences remained substantial at larger sizes.
- 3.3. PeptideCLM-2 adaptation is less influential than training-set size: PeptideCLM-2 adaptation strategies differed little relative to training-size effects, showed unstable ordering across sizes, and had no overall AULC difference.At 6,000 records, mean SPCC ranged from 0.452 to 0.484; the Friedman test gave χ2 = 6.62 and p = 0.251.
- 3.4. Explicit pairwise features provide no consistent gain: Interaction-aware fusion provided no consistent gain over direct concatenation across peptide-similarity thresholds.Concatenation was higher at seven thresholds and interaction-aware fusion at three; the paired contrast ranged from -0.0079 to +0.0061.
4. Discussion
The benchmark shows that affinity-model performance and representation rankings depend on the generalization regime, training-data scale, and downstream learner. Its conclusions are bounded by the pooled dataset, exact-sequence target exclusion, and unmodeled assay and structural factors.
- ECFP-16 count fingerprints with random forest led peptide-similarity and within-target evaluation, while HELM-BERT with Extra Trees led leave-target-out evaluation.
- Interaction-classification performance cannot be assumed to imply accurate affinity ranking for an unseen target because this benchmark evaluates quantitative affinity under controlled peptide and target shifts.
- Representation differences were largest under low-data conditions and narrowed as supervision increased, whereas tree ensembles continued improving at larger training sizes.
- Element-wise products and absolute differences did not reliably improve prediction within the independent-encoding and MLP framework, but this does not exclude richer interaction architectures.
- The inference scope is limited by pooled Kd, Ki, and IC50 measurements, exact-sequence rather than homology-aware target exclusion, canonical-peptide predominance, and no independent external dataset.
- Representation superiority is conditional on the intended generalization regime, so benchmarks should align partitions with use, compare multiple learners, and report sensitivity to data availability.