PaperlayerView exact revision on arXiv

Source-linked AI summary

Are You Learning Biological Signal or Shortcuts? Auditing and Mitigating Bias in Protein-Protein Interaction Datasets

Judith Bernett, Anton Spannagl, Joel Ås, Markus List, David B. Blumenthal

arXiv:2609.10193v1q-bio.MNcs.LG

TL;DR

PPI datasets contain study and technical biases that can let models learn shortcuts rather than biological interaction signal, but the construction-related shortcuts had not been systematically examined together. The paper audits these biases across multiple PPI sources and provides integer-linear-program methods for similarity-aware splitting and bias-minimizing negative sampling. It finds that shortcuts persist after overlap removal, vary by data source, and can be amplified by high-confidence non-interactor sampling.

  • Problem

    PPI databases contain study and technical biases, and careless negative sampling can create distributional differences that allow models to learn shortcuts instead of biological interaction properties.

  • Method

    The paper audits shortcut attributes across PPI datasets and provides an open Nextflow pipeline using integer linear programs for data-loss-minimizing splitting and bias-minimizing negative sampling.

  • Results

    Shortcuts vary across databases: topology is strong under random splitting, while self-interactions, taxonomy, and functional relatedness remain after overlap removal; high-confidence negatives can amplify functional-relatedness bias.

  • Takeaways & Limitations

    Bias auditing and optimization-based dataset construction are practical tools for reducing shortcut learning in PPI prediction and may extend to other problems with many negative candidates.

  • Takeaways & Limitations

    Structural datasets contain many positive self-interactions, making it impossible for the negative sampler to match positive and negative self-interaction counts.

Abstract

from arXiv · show

Protein-protein interaction (PPI) databases do not faithfully reflect biological realities. Instead, they are influenced by study and technical biases that distort certain protein and interaction attributes. Machine learning models can exploit these as learning shortcuts if the negative dataset is not constructed with care. So far, the shortcuts introduced during PPI dataset construction have only been examined in isolation. Here, we systematically characterize both reported and, to our knowledge, previously unreported biases in PPI datasets that lead machine learning models to learn shortcuts instead of biological signal. We analyze HIPPIE, IntAct, and STRING, dedicated PPI databases, as well as two datasets derived from 3D-structural information in the Protein Data Bank (PDB). We show that random data splitting introduces strong topological shortcuts. When train-test protein overlap is removed, the resulting datasets still retain usable shortcuts stemming from self-interactions, taxonomic identity, and functional relatedness, whose prevalence interestingly depends on the data source. We further show that sampling negatives from a set of high-confidence non-interactors, an intuitively appealing choice, can amplify the shortcut stemming from functional relatedness. To detect and mitigate these biases, we provide an open Nextflow pipeline that combines similarity-aware, data-loss-minimizing dataset splitting with bias-minimizing negative sampling, both formulated as integer linear programs. Its key concept of quantifying biases to minimize them through optimization-based negative sampling can, in principle, be extended to any machine learning problem where the pool of negative candidates is much larger than the positives and is thus of interest also beyond PPI prediction.

1 Introduction

PPI prediction is vulnerable to shortcut learning because biased databases and careless negative sampling can create distributional differences that models exploit instead of biological interaction signals. This paper systematically audits these biases and introduces optimization-based methods to detect and mitigate them.

  • Motivation: Understanding model reliance is especially important in biomedical applications because humans cannot easily verify whether predictions reflect appropriate high-level abilities.The paper contrasts this difficulty with domains such as object detection and text translation.
  • Motivation: PPI prediction models may exploit study and technical biases in interaction databases instead of learning biochemical properties such as binding-site compatibility.These biases include overstudy of human disease-related proteins and technical effects such as low abundance.
  • Motivation: Randomly sampled negatives can create distributional differences from positives, enabling machine learning shortcuts rather than biologically meaningful predictions.Previously reported shortcut attributes include subcellular location, sequence similarity, and topology.
  • Contributions: The study quantifies shortcut potential across self-interactions, taxonomy, topology, sequence and embedding similarity, and Gene Ontology overlap using G-AUDIT.G-AUDIT measures attribute detectability from inputs and mutual information with the output label.
  • Contributions: The authors formulate integer linear programs for similarity-aware, data-loss-minimizing splitting and bias-minimizing negative sampling.The negative sampler matches shortcut-attribute distributions between sampled negatives and annotated positives.
  • Contributions: The analyses are released as an open Nextflow pipeline that the community can apply to its own datasets.The pipeline supports the paper’s auditing and mitigation workflow.

2 Results

Across PPI datasets, models can exploit topological, self-interaction, taxonomic, functional, and residual distributional shortcuts rather than biological interaction signal. ILP-based splitting reduces overlap-related shortcuts, while bias-minimizing negative sampling further lowers performance toward chance but cannot eliminate constraints imposed by the data source.

  • Performance under the maximally biased setting: Topology is the strongest detectable shortcut across datasets, with protein degree differences making several maximally biased datasets trivially separable.Removing train-test protein overlap makes topology unavailable outside the maximally biased setting.
  • Performance under the maximally biased setting: Self-interactions dominate PDB-Dimers and PINDER, comprising 80% and roughly 40% of positive interactions, respectively.Random negatives rarely contain self-pairs, making protein identity highly predictive; self-interactions also induce same-species, embedding-similarity, and GO-overlap shortcuts.
  • Performance under the maximally biased setting: About 90% of annotated IntAct, PDB-Dimers, and PINDER interactions are within species, compared with fewer than 20% of random negatives, creating a taxonomy shortcut.Species is reasonably inferable from embeddings, although inter-species interactions remain biologically possible.
  • Performance under the maximally biased setting: GO-category overlap is especially predictive in STRING-900, STRING-database, and STRING-textmining because these subsets are functionally enriched.Selecting high-confidence STRING interactions can therefore make prediction easier than on purely experimental data.
  • Similarity-reduced versus ILP-based splitting: Splitting alone leaves structural-dataset AUPRC at 0.95–0.99 and cannot remove self-interaction, taxonomy, or GO-overlap shortcuts arising from positive-negative distribution differences.PDB-Dimers and PINDER retain high performance because self-interactions cannot be removed by splitting.
  • Controlled bias-reduction through the ILP-based negative sampling: ILP sampling reduces GO-overlap mutual information but cannot eliminate it in STRING-900 and STRING-database because few functionally related high-confidence negatives remain available.For several other datasets, all four modeled bias terms are reduced to near zero.
  • Controlled bias-reduction through the ILP-based negative sampling: Balanced AUPRC falls to 0.53–0.58 and realistic AUPRC to 0.08–0.13 after ILP-based sampling for all datasets except PDB-Dimers and PINDER.ILP-based sampling yields the lowest performance on every dataset except HIPPIE’s balanced test set, where it reaches 0.56 versus 0.54 for ILP splitting alone.

3 Discussion

Splitting and negative-sampling choices determine which shortcuts PPI models can exploit, with shortcut magnitude varying across databases.

  • Removing train-test protein overlap and minimizing inter-split sequence similarity counteract shortcuts, but distributional differences between positive and negative sets remain.
  • The ILP-based negative sampler reduces the baseline to chance across all datasets except PDB-Dimers and PINDER, where self-interactions remain the largest issue.
  • Random splitting can yield near-perfect Random Forest performance by exposing topological shortcuts rather than biological interaction signal.
  • Residual shortcuts differ by source: PDB-derived datasets retain self-interaction bias, multi-species datasets retain taxonomic cues, and some STRING subsets retain functional enrichment cues.

B HIPPIE STRING human, physical

High-confidence negative sampling can worsen distributional shifts, especially in STRING, while the paper emphasizes that splitting and sampling choices determine how PPI performance should be interpreted. The released pipeline quantifies and minimizes such shortcuts, but the work is not a benchmark of state-of-the-art models.

  • B HIPPIE STRING human, physical: The choice of splitting and sampling strategy is essential for interpreting reported PPI performance and generalization to unseen proteins.A high AUPRC from random splitting and uniform negative sampling says little about generalization to unseen proteins.
  • B HIPPIE STRING human, physical: Database choice independently affects results, with PDB-derived datasets representing a population heavily weighted toward homo(di)mers.This imbalance limits their direct use for augmenting PPI datasets with structural information.
  • B HIPPIE STRING human, physical: The analysis is technically limited by support for only ESM-2 and ProtT5 embeddings, a non-exhaustive attribute set, and optimization over a sampled candidate pool.The returned negative set is optimal for that sampled pool, although a better-matched set may exist.
  • B HIPPIE STRING human, physical: The open pipeline quantifies shortcuts and minimizes them through optimization-based negative sampling, with bias terms optionally disabled when treated as signal.The approach is designed to extend to other pair-input and binary classification problems with sufficiently large negative candidate pools.
  • B HIPPIE STRING human, physical: The study is not a benchmarking paper and leaves comparison of state-of-the-art deep PPI models on de-biased splits to future work.Its focus is identifying dataset biases and producing de-biased splits for training and evaluation.

4 Methods

The methods compare PPI datasets across evidence sources and construct bias-aware splits, negative samples, and bias measurements. The pipeline combines similarity reduction, integer-linear optimization, and attribute-based shortcut analysis.

  • 4.1 Datasets: The study analyzes experimentally curated, STRING-derived, and structure-inferred PPI datasets spanning species scope, evidence type, and detection modality.The structural datasets are PINDER and PDB-Dimers, while HIPPIE, IntAct, and STRING represent interaction databases.
  • 4.1 Datasets: PINDER retains interactions between chains within 10 Å in biological assemblies, whereas PDB-Dimers restricts assemblies to exactly two protein-chain instances.PDB-Dimers is designed to reduce dependence on additional chains in higher-order complexes.
  • 4.2 The PPI splitting and bias analysis pipeline: The pipeline takes UniProt-based PPI CSV files and returns train, validation, and test sets, sampled negatives, bias metrics, baseline performance, and an interactive report.It also fetches sequences, taxonomy, and Gene Ontology annotations and can perform bias-reducing negative sampling.
  • 4.2.1 Splitting of the positives: Random splitting assigns PPIs to partitions without constraints, while KaHIP minimizes sequence similarity by placing proteins into three blocks and retaining within-block interactions.The largest, second-largest, and smallest blocks become training, validation, and test sets, respectively.
  • 4.2.1 Splitting of the positives: The ILP split assigns whole KaHIP similarity clusters to partitions while minimizing data loss and enforcing target fractions over retained PPIs.Auxiliary variables linearize cluster-pair retention, and PPIs within the same cluster are never discarded.
  • 4.2.4 Bias analysis: Bias analysis treats an attribute as shortcut-relevant when it is detectable from the input and informative about the output label.The study uses normalized mutual information for utility and embedding-based Ridge prediction with Spearman correlation for detectability.

7 Broader impact statement

The study uses public biological databases containing reference sequences and annotations rather than individual-level human data. Its intended benefit is more informative benchmarks, while its main stated risk is over-interpretation.

  • 7 Broader impact statement: The analyses use public biological databases and do not involve personally identifiable information.The sources include UniProt, IntAct, STRING, PDB, and Gene Ontology.
  • 7 Broader impact statement: More informative benchmarks may reduce misdirected methodological effort and wasted laboratory resources when models prioritize experiments.The paper identifies over-interpretation of datasets as its main risk.

A Additional results for ESM-2 embeddings

The supplementary ESM-2 results provide additional comparisons across dataset splitting strategies, test sets, and performance metrics. They also report data retention for the proposed splitting approaches.

  • A Additional results for ESM-2 embeddings: Figure A1 compares the percentage of identical residues between interaction candidates in the training datasets.The passage identifies the sequence-identity comparison but does not provide numerical values.
  • A Additional results for ESM-2 embeddings: Table A1 reports performance metrics across datasets, splitting strategies, and test sets for the ESM-2-based analyses.The table continues across multiple supplementary panels.
  • A Additional results for ESM-2 embeddings: Figure A2 compares data retained for training, validation, and test with data discarded by the two suggested splitting approaches.CD-HIT-2D removes 22 380 IntAct interactions and 3096 PINDER interactions in the similarity-reduced setting, versus 17 951 and 1838 under the ILP split.

B Results for ProtT5 embeddings

The ProtT5 supplementary results compare embedding utility, bias detectability, and baseline classifier performance with the corresponding ESM-2 analyses. Overall, utility is comparable, biases are more detectable from ESM-2, and ProtT5 baseline results are better.

  • B Results for ProtT5 embeddings: The utility of the embedding-similarity attribute is overall comparable between ESM-2 and ProtT5.Figure B1 reports the cross-embedding comparison.
  • B Results for ProtT5 embeddings: Biases are overall more detectable from ESM-2 than from ProtT5 embeddings in the maximally biased setting.Figure B2 compares detectability using concatenated embeddings and a Ridge regressor.
  • B Results for ProtT5 embeddings: ProtT5 baseline Random Forest results are better overall than the corresponding ESM-2 results.Figure B3 presents the comparison without specifying metric values in the supplied passage.
  • B Results for ProtT5 embeddings: Tables B1 and its continuation report performance metrics across datasets, splitting strategies, and test sets for ProtT5 embeddings.The supplied captions identify the scope but do not provide cell-level values.

C Runtime and resource consumption

Pipeline runtime is concentrated in bias analysis, ILP solving, clustering, and embedding computation, while memory peaks during baseline training and bias analysis. The full set of analyses completed in about 3 days and 17 hours, with substantial variation across pipeline configurations.

  • Runtime: The bias analysis, ILP solving, KaHIP clustering, and embedding computation were the pipeline steps with the longest runtimes.KaHIP clustering into 100 clusters is required only for the ILP split.
  • Runtime: The positive-split ILP always reached the 2-hour time limit, producing feasible but not necessarily optimal cluster assignments.This makes the reported assignments feasible rather than guaranteed optimal.
  • Runtime: Negative-sampling ILP runtime approached 6 hours on the training set because it scales quadratically with the candidate pool.The pool was therefore capped at four times the positive count, about 2 million candidates for training.
  • Overall resource use: The full pipeline completed in about 3 days and 17 hours across all 43 settings, given the available concurrency.The reported settings included nine datasets in four configurations plus high-confidence non-interactor sampling for seven datasets.
  • Memory: Memory consumption was highest for baseline training and bias analysis, while all other pipeline steps stayed below 15 GB.Baseline training stores concatenated embeddings for all pairs, and the largest datasets had the highest memory use.
Loading 2609.10193v1…