PaperlayerView exact revision on arXiv

Source-linked AI summary

CaliPPer: quantifying, predicting and improving AI model performance for binding prediction

Jian-Qing Zheng, Hantao Lou, Zinan Yin, Sam Farrar, Yuze Zhou, Elie Antoun, Xiangxi Wang, Xuetao Cao, Tao Dong

arXiv:2606.07258v1cs.CEcs.AIcs.LGq-bio.QM

TL;DR

Binding models can perform well on familiar data but their performance on new targets is difficult to predict, while existing density-ratio estimators rely on assumptions often violated by novel epitopes and variants. CaliPPer combines multi-chain distance with distance-aware Bayesian recalibration to estimate performance at model, dataset, and sample levels. Across benchmark and retrospective studies, it predicted performance and improved discrimination and discovery outcomes.

  • Problem

    Binding models’ performance on new epitopes and antigen variants is unpredictable, and density-ratio estimators assume shared sequence-space support and unchanged binding rules that novel targets violate.

    Source: Main · pp. 1–2

  • Method

    CaliPPer combines multi-chain Sample-to-Domain Distance with distance-aware performance estimation and Bayesian recalibration at model, dataset, and sample levels.

    Source: Main · p. 2, Discussion · p. 6

  • Results

    Across five retrospective studies, recalibration improved discovery outcomes, including confirmed deepAntigen neoantigens rising from 0/5 to 3/5; validation spanned TCR, BCR, MHC–peptide, and small-molecule tasks.

    Source: Discussion · p. 6

  • Takeaways & Limitations

    Distance-aware recalibration can improve candidate rankings in the reported discovery settings, yielding roughly 1–2 additional confirmed hits per 10 candidates tested in the deepAntigen pipeline.

    Source: Discussion · p. 6

    Reliable curves require at least 30 calibration samples per distance bin, at least 8 from the smaller class, and query distances within the calibration range; extrapolation reduces reliability.

    Source: Discussion · p. 7

Abstract

from arXiv · show

Binding prediction models accelerate therapeutic antibody and TCR discovery, but their performance on new datasets is unpredictable, often leading to low discovery rates. Density-ratio methods (PAPE, M-CBPE) provide label-free performance estimation for binary classification, but their assumptions and aggregate-only outputs limit binding prediction on neoepitopes, antigen variants and chemical scaffolds. Here we present CaliPPer (Calibration and Prediction of Performance), a post-hoc framework pairing a multi-chain Sample-to-Domain Distance (S2DD) with distance-aware Bayesian recalibration, operating at three resolutions: generalisability score, aggregate performance prediction, and per-sample confidence. Across ten models, eight architectures and two immune-receptor domains, CaliPPer attains distance--performance correlations $|r|=0.80\text{--}0.92$, predicts AUROC/AP/F1 with mean absolute errors $0.008\text{--}0.070$, and improves AUROC by up to $+0.20$ on unseen epitopes/variants. Applied retrospectively to five published TCR, BCR, MHC--peptide and small-molecule studies, CaliPPer raises true discovery rates in all five (e.g.\ $0/5 \to 3/5$ confirmed neoantigens), providing a triage layer between computational prediction and experimental validation.

Main

Binding models can perform well on familiar benchmarks yet fail unpredictably on new epitopes and antigen variants. CaliPPer uses sequence-space distance to estimate generalisability, predict performance on unlabelled cohorts, and recalibrate individual predictions.

Motivation

  • 23 of 50 binding models achieved AUPRC ≤0.5 on held-out epitopes, while BCR models also show a generalisation gap across antigen variants and pathogen classes.Failed binding validations consume reagents and bench time and can delay therapeutic development.Source: Main · p. 1

Limits of existing estimators

  • PAPE and M-CBPE assume training and test data share sequence-space support and binding rules, assumptions novel epitopes and antigens violate.These methods can collapse toward a single average estimate across test sets and return no per-sample confidence for candidate prioritisation.Source: Main · p. 2

Approach

  • CaliPPer models performance as a function of Sample-to-Domain Distance (S2DD), combining receptor and antigen distances with weights that emphasize concentrated and informative chains.Its distance component can use edit distance, BLOSUM scores, ESM-2 embeddings, or structural RMSD formulations.Source: Main · p. 2

Three operating resolutions

  • CaliPPer provides distance-resolved model generalisability, aggregate performance estimates for unlabelled data, and distance-aware per-sample confidence recalibration.The data-level estimate combines an S2DD-conditioned degradation curve with a PAPE-derived density-ratio correction; per-sample recalibration uses distance-dependent PPV and NPV.Source: Main · p. 2

Validation overview

  • Across ten models in TCR and BCR binding, CaliPPer predicted held-out performance with MAEs of 0.008–0.070 and improved AUROC by up to +0.20 without retraining.Retrospective application to five studies improved true discovery rates in all five, including neoantigen candidates from 0/5 to 3/5 confirmed.Source: Main · p. 2

Results

Across binding models and targets, performance generally declines as S2DD distance from training increases, and the relationship supports aggregate estimates, per-target predictions, and per-sample recalibration. Retrospective applications also improved discovery outcomes across five published studies.

S2DD reveals consistent performance degradation across architectures and domains

  • Across ten models, S2DD correlated negatively with performance; TCR cross-validation |r| was 0.81–0.92 and BCR |r| was 0.80–0.88, while multi-chain S2DD outperformed a single-chain TCR baseline (mean |r| 0.87 vs 0.43).The distance–performance pattern also held on cross-test data, including unseen epitopes; BCR AUROC correlations were weaker where performance varied over a compressed range.Source: S2DD reveals consistent performance… · p. 3

CaliPPer predicts dataset-level performance on unlabelled data

  • Dataset-level predictions tracked observed performance across four evaluation settings, with MAEs of 0.008–0.070; at nearest distances, error without distance correction was 4.9× higher for AUROC and 4.6× for AP.BCR cross-test correlation was lower (R = 0.407) where the AUROC range was narrow, although MAE remained 0.070.Source: CaliPPer predicts dataset-level… · pp. 3–4

CaliPPer predicts per-epitope and per-variant performance

  • Per-target predictions correlated with observed performance for TCR epitopes (r = 0.78 for AP; r = 0.71 for AUROC) and BCR variants (r = 0.77 for AP; r = 0.65 for AUROC).In TCR epitope-split prediction, median |r| was 0.83 with CaliPPer versus 0.73 for PAPE and 0.69 for M-CBPE; BCR gains were small, and TCR prediction error increased on lower-AUROC targets.Source: CaliPPer predicts per-epitope and… · p. 4

CaliPPer recalibration improves prediction reliability

  • Recalibration improved AUROC and AP in most model–fold combinations, with the largest gains on out-of-distribution data, including +0.066 mean AUROC on unseen TCR epitopes and +0.046 on BCR influenza.Gains scaled positively with S2DD distance in TCR (r = 0.939) and BCR (r = 0.790); near-distance TCR AUROC changed by about −0.003 on average.Source: CaliPPer recalibration improves… · pp. 4–5

Independent retrospective validation across five published studies

  • Across five retrospective studies, recalibration improved AUROC and AP in all five; deepAntigen neoantigen confirmations rose from 0/5 to 3/5, and BigMHC TDR at k=50 rose from 22/50 to 29/50.CaliPPer also achieved lower average dataset-level prediction errors than PAPE and M-CBPE, including in the small-molecule study where sequence-based density ratios do not apply.Source: Independent retrospective validation… · pp. 5–6

Discussion

CaliPPer combines distance-based performance estimation and recalibration to support model comparison, cohort assessment, and candidate triage. Its retrospective results span diverse tasks, while its reliability depends on calibration data covering relevant distance regimes and sample requirements.

Discussion

  • Its three output levels support distance-aware model choice, cohort performance estimates, and per-candidate confidence for experimental prioritisation.The discussion also recommends reporting degradation curves alongside aggregate performance and using chain-concentration analysis to inform training-data design.Source: Discussion · p. 6
  • CaliPPer combines multi-chain S2DD, a distance-anchored correction to density-ratio estimates, and Bayesian recalibration that can rerank candidates without retraining.Recalibration uses distance-dependent PPV and NPV; when model scores are already accurate it behaves like simple rescaling, and when they are unreliable it leans on distance.Source: Discussion · p. 6
  • 0 of the top 5 raw-score deepAntigen candidates were confirmed immunogenic, versus 3 of 5 after CaliPPer recalibration; true discovery rates rose across all five retrospective studies.At the top of the ranking, the paper reports roughly 1–2 additional confirmed hits per 10 candidates tested.Source: Discussion · p. 6
  • CaliPPer’s patterns held across TCR and BCR regimes differing by a 20-percentage-point gap in median AUROC, and predicted near-chance TCR values rather than defaulting to 0.5.Validation also covered ten models, eight architectural families, and retrospective TCR, BCR, MHC–peptide, and small-molecule studies.Source: Discussion · pp. 6–7
  • Reliable use requires at least 30 calibration samples per distance bin, query distances within the calibration range, and a representative non-flat PPV/NPV profile.If calibration data are too concentrated or fall entirely on a flat performance plateau, recalibration reduces to standard Platt scaling and leaves predictions unchanged; existing held-out validation cohorts can suffice when they cover novel targets.Source: Discussion · p. 7
  • The authors identify HLA class II–restricted CD4+ TCR–peptide benchmarking as a next direction because their TCR data are predominantly HLA class I-restricted.They also list extensions including model-internal uncertainty, active learning, continuous-affinity prediction, and alternative distance types.Source: Discussion · p. 7

Methods

CaliPPer uses a shared multi-chain distance signal to produce model-, dataset-, and sample-level reliability outputs. Its post-hoc Bayesian recalibration adjusts per-sample predictions using distance-dependent calibration estimates.

Framework Design

  • The framework pairs multi-chain Sample-to-Domain Distance (S2DD) with distance-aware Bayesian recalibration to evaluate, predict, and improve binary binding-model reliability.S2DD aggregates LogDist across receptor and antigen chains using concentration-based weighting, then applies top-K averaging to produce a scalar per sample.Source: Framework Design · p. 15
  • The shared S2DD computation yields a model-level generalisability slope, dataset-level aggregate performance predictions, and sample-level calibrated binding probabilities, all post-hoc on any trained binary classifier.Aggregate predictions use bias-corrected decay curves; sample-level recalibration derives Platt-scaling parameters analytically from distance-dependent PPV and NPV estimated on labelled calibration data.Source: Framework Design · p. 15

Problem Formulation

The formulation defines a reference training set, a new query set, and model predictions with unobserved binary labels. It frames CaliPPer around estimating distance, predicting aggregate performance without test labels, and recalibrating individual predictions.

Problem Formulation

  • Each query sample has C molecular components, a model prediction, and an unobserved binary ground-truth label, relative to training set R and test set Q.TCR and BCR examples each use three components, while small-molecule applications substitute molecular fingerprint feature vectors.Source: Problem Formulation · p. 15
  • The distance task seeks a scalar query-to-training distance whose binned performance correlation is negative and whose distance–performance relationship generalises across datasets, splits, and models.Source: Problem Formulation · p. 15
  • The other tasks predict AUROC, AP, or F1 on an unlabelled query set and recalibrate each sample’s probability using both its raw prediction and distance.Aggregate prediction accuracy is measured by mean absolute error; recalibration quality is assessed by discrimination improvement against raw predictions.Source: Problem Formulation · p. 15

S2DD

S2DD converts multi-chain sequence comparisons into a per-sample distance from the training distribution. Its hierarchy combines chain-level distances, selects the most outlying weighted chain for each sample pair, and averages over nearby references.

Per-Pair Distance

  • Logarithmic transformation expands the sparse, near-training tail of right-skewed raw distances, sharpening discrimination where degradation is most informative.The square-root transformation is appropriate for more evenly populated raw distances and avoids the log(0) singularity at sim = 1.Source: Per-Pair Distance · pp. 15–16

Multi-Chain Combination

  • For each query-reference pair, S2DD combines per-chain distances by selecting the chain with the largest weighted z-score, yielding an unscaled z-score.The selected chain makes the combined distance large when a sufficiently weighted chain is far from the training distribution.Source: Multi-Chain Combination · p. 16

Chain Weighting

  • Chain weights are computed automatically from training-set distance variability and sequence concentration, without user-set hyperparameters.The peptide chain dominated for the evaluated TCR datasets and the antigen-variant chain for the evaluated BCR datasets; the authors caution that these patterns reflect those training distributions, not general biological properties.Source: Chain Weighting · pp. 16–17

Sample-to-Distribution Reduction

  • S2DD reduces a query’s distances to all references by averaging its distances to the K nearest references.The paper uses K = 50 for TCR data and K = 30 for BCR data; these references are jointly close across all chains.Source: Sample-to-Distribution Reduction · p. 17

S2DD Summary

  • S2DD provides a three-level pipeline from chain-to-chain comparisons, through sample-to-sample distances, to a per-sample distance from the training distribution.Its base distance is modular; the paper uses BLOSUM for TCR results and Levenshtein for BCR results, with the choice guided by sequence length.Source: S2DD · p. 15, S2DD Summary · p. 17

Binned Evaluation and Correlation Analysis

The evaluation bins query samples by increasing S2DD distance, then measures how performance changes across bins. Correlations assess degradation trends, while a regression slope quantifies their rate.

Binned Evaluation and Correlation Analysis

  • Query samples are sorted by S2DD distance and divided into equal-sized bins, with the bin count adaptively set from the minority-class count.The procedure chooses B = max(4, min(8, floor(nmin/8))) to balance resolution against statistical power.Source: Binned Evaluation and Correlation… · p. 18
  • Pearson correlation between bin distance and bin performance measures the linear degradation trend; a strongly negative, significant r indicates performance tends to decline as distance increases.Spearman correlation additionally assesses monotonic, potentially nonlinear trends and is robust to outlier bins; agreement between r and ρ strengthens confidence in the trend.Source: Binned Evaluation and Correlation… · p. 18
  • The regression slope γ estimates performance change per unit increase in S2DD, with a more negative value indicating faster degradation.It complements correlation magnitude: models with similar correlations can have different degradation rates.Source: Binned Evaluation and Correlation… · p. 18

CaliPPer Performance Prediction

CaliPPer predicts aggregate performance on unlabelled datasets by combining distance-conditioned performance curves with density-ratio estimates. The bias-corrected curve improves AUROC prediction over a plain exponential curve in the reported TCR cross-test comparison.

CaliPPer Performance Prediction

  • CaliPPer fits an exponential decay curve to binned distance-performance data from known evaluation settings, enforcing non-increasing performance with distance.The exponential model is selected for its interpretability and three parameters among linear, exponential, and isotonic candidates.Source: CaliPPer Performance Prediction · p. 18
  • A unified bias-corrected curve adds each bin’s mean predicted probability to account for variation in calibration and class balance across evaluation settings.For a new dataset, CaliPPer evaluates the curve at each bin’s mean distance and mean predicted probability, then aggregates bin predictions by sample count.Source: CaliPPer Performance Prediction · pp. 18–19
  • AUROC prediction R = 0.75 vs. 0.56 for the bias-corrected curve versus the plain exponential on TCR cross-test.The reported improvement is +0.19 R; the bias term captures calibration variation across datasets.Source: CaliPPer Performance Prediction · p. 19
  • The density-ratio-corrected estimate calibrates per-sample predictions under the target distribution, then fits a distance-based residual correction to capture between-test-set calibration shift.Exponential and right-side Gaussian residual curves are compared, with a parsimony threshold favoring the simpler exponential when fits are sufficiently similar.Source: CaliPPer Performance Prediction · p. 19
  • Leave-one-out cross-validation and cross-test evaluations assess aggregate prediction error using MAE between predicted and actual performance for each held-out setting.This MAE measures error in one overall-performance scalar per held-out dataset, not similarity between degradation-curve profiles.Source: CaliPPer Performance Prediction · p. 19

CaliPPer Bayesian Recalibration

CaliPPer’s recalibration uses distance-dependent information to alter both probability calibration and ranking. It derives Platt-scaling parameters analytically from distance-specific predictive values rather than fitting them by maximum likelihood.

CaliPPer Bayesian Recalibration

  • Unlike standard monotone Platt scaling, CaliPPer derives distance-dependent Platt parameters analytically from PPV(d) and NPV(d), allowing recalibration to adjust ranking as well as probabilities.The per-bin slope-weighting factor gates the contribution of the parameters relative to the prior.Source: CaliPPer Bayesian Recalibration · p. 19
  • The posterior combines the raw prediction, calibration prevalence, and distance-dependent Platt parameters; no MCMC sampling is required.The parameters are obtained in closed form from per-bin PPV and NPV, with positive-class prevalence as the only prior.Source: CaliPPer Bayesian Recalibration · p. 19
  • Because the Platt parameters vary with S2DD distance, samples with identical raw scores can receive different posterior probabilities.The algorithm applies a weighted calibrator to calibration labels before estimating aggregate performance on test predictions.Source: CaliPPer Bayesian Recalibration · p. 20

12 end

CaliPPer fits distance-aware PPV and NPV curves from independently binned calibration sets, then uses them to recalibrate test predictions. Adaptive thresholds and distance-dependent slope weighting address class imbalance and preserve within-bin ordering.

12 end

  • The adaptive classification threshold equals 0.5 when calibration prevalence exceeds 0.25 and adjusts downward under more extreme imbalance to avoid degenerate PPV estimates.The threshold defines positive and negative predictions used to estimate PPV and NPV.Source: §12 · p. 20
  • Distance-dependent PPV and NPV curves are fitted to per-bin mean distance and prediction, with β capturing between-test-set variation preserved by binning each calibration set independently.Pooling calibration sets before binning destroys that signal and degrades calibration; flat distance curves reduce the procedure to standard Platt scaling.Source: §12 · pp. 20–21
  • Slope weighting attenuates calibration where PPV+NPV indicates weak discrimination, preventing within-bin rank inversion while retaining distance-based reordering across bins.When discrimination is below random, the floor shrinks recalibration toward the prior rather than amplifying a rank-inverting update.Source: §12 · pp. 20–21

Datasets

  • Algorithm 3 fits PPV and NPV curves from distance-ordered calibration bins, evaluates them for test bins, and produces distance-dependent recalibrated probabilities.It uses calibration prevalence and an adaptive threshold; test samples receive estimates based on their bin’s mean prediction.Source: Datasets · p. 22

28 end

The evaluation uses established TCR and BCR split conventions, ten models, and post-hoc application to five retrospective studies. Statistical reporting includes paired recalibration tests, while pooled analyses treat nested observations as independent for descriptive summaries.

28 end

  • TCR evaluation includes unseen-epitope and external cross-tests, while BCR evaluation tests unseen antibodies and antigen variants; five retrospective studies use published partitions and disjoint calibration and test sets.For PanPep and AntibioticsAI, a distance-sorted half of the test cohort serves as calibration; per-receptor fine-tuning studies such as PRP-TCR were excluded as a different setting.Source: §28 · p. 23

Models

  • Ten primary models span eight architectural families across TCR and BCR, and five retrospective models use published weights and pipelines without retraining or fine-tuning.Retrospective modalities include TCR–peptide, BCR–antigen, MHC–peptide, and small-molecule chemistry.Source: Models · p. 24

Evaluation Protocol

  • Retrospective S2DD distances are chosen to match each study’s input modality, and AUROC and AP are reported alongside top-k true-discovery rates for prioritisation studies.XBCR-net instead uses a reranking scatter for its 21-candidate panel and reports TDR at k = 8.Source: Evaluation Protocol · pp. 24–25

Statistical analysis

  • Paired recalibration comparisons use one-sided Wilcoxon tests, but pooled summaries treat nested observations as independent and do not formally model their structure.TCR-unseen AUROC and AP comparisons reached significance; TCR-seen and BCR per-domain comparisons did not reach p < 0.05.Source: Statistical analysis · p. 25

Methods References

The references include work on T-cell specificity prediction, antibody databases, and influenza antibody studies.

Methods References

  • Lu et al. describe deep-learning prediction of T-cell receptor–antigen binding specificity [36].Source: Methods References · p. 26
  • CoV-AbDab is cited as a coronavirus antibody database [38].Source: Methods References · p. 26
  • The listed works [39–44] concern vaccine-induced antibodies neutralizing group 1 and 2 influenza A viruses, cross-lineage influenza B hemagglutinin protection, immune-history effects, elderly responses, antibody landscapes, and H5N1 stem antibodies.Source: Methods References · p. 26

Data Availability

The paper identifies public and request-access datasets used in its analyses and states that processed datasets and evaluation splits are available in the code repository.

Data Availability

  • TCR data include publicly available 10x Genomics datasets and VDJdb, while in-house paired SARS-CoV-2 TCR data are available from the corresponding author upon reasonable request.Source: Data Availability · pp. 26–27
  • External TCR cross-test sets came from McPAS-TCR and IEDB, and antibody and retrospective-study data were obtained from CoV-AbDab and the original publications.Source: Data Availability · p. 27
  • Processed datasets, with in-house data replaced by anonymised sequence hashes where applicable, and evaluation splits are available in the code repository.Source: Data Availability · p. 27

Author contributions statement

J.Q.Z. and H.L. led conceptualization, methodology, software, validation, and the original draft; contributions across investigation, analysis, data, visualization, writing, resources, and supervision involved the wider team.

Author contributions statement

Loading 2606.07258v1…