PaperlayerView exact revision on arXiv

Source-linked AI summary

PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding

Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Chang Ma, Runcheng Liu, Jian Tang

arXiv:2206.02096v2cs.LG

TL;DR

Protein deep learning lacks a standard benchmark for comparing methods across diverse tasks. PEER fills this gap with a comprehensive multi-task benchmark for protein sequence understanding and evaluates several sequence-based model families. Pre-trained protein language models perform best on most individual tasks, and joint multi-task training can further improve performance.

  • Problem

    Protein deep learning methods lack a standard benchmark for systematic evaluation across protein understanding tasks.

  • Method

    PEER benchmarks sequence-based approaches across 14 tasks spanning protein function, localization, structure, protein-protein interaction, and protein-ligand interaction prediction, including multi-task learning.

  • Results

    Pre-trained protein language models achieve the best performance on most individual tasks, while jointly training multiple tasks further enhances performance.

  • Takeaways & Limitations

    PEER provides a common basis for comparing protein sequence methods and indicates that model pre-training and suitable multi-task training are promising across tasks.

  • Takeaways & Limitations

    The benchmark remains limited in scope and is planned for expansion with more tasks, datasets, and baseline models.

Abstract

from arXiv · show

We are now witnessing significant progress of deep learning methods in a variety of tasks (or datasets) of proteins. However, there is a lack of a standard benchmark to evaluate the performance of different methods, which hinders the progress of deep learning in this field. In this paper, we propose such a benchmark called PEER, a comprehensive and multi-task benchmark for Protein sEquence undERstanding. PEER provides a set of diverse protein understanding tasks including protein function prediction, protein localization prediction, protein structure prediction, protein-protein interaction prediction, and protein-ligand interaction prediction. We evaluate different types of sequence-based methods for each task including traditional feature engineering approaches, different sequence encoding methods as well as large-scale pre-trained protein language models. In addition, we also investigate the performance of these methods under the multi-task learning setting. Experimental results show that large-scale pre-trained protein language models achieve the best performance for most individual tasks, and jointly training multiple tasks further boosts the performance. The datasets and source codes of this benchmark are all available at https://github.com/DeepGraphLearning/PEER_Benchmark

1 Introduction

Protein deep learning methods span multiple tasks but lack a standard benchmark for systematic comparison. PEER addresses this gap with diverse tasks, carefully designed splits, and evaluations of sequence-based and multi-task methods.

  • 1 Introduction: Existing protein deep learning studies use varied sequence encoders but are typically evaluated on different tasks and datasets, preventing systematic comparison.The cited encoders include CNNs, LSTMs, and Transformers.
  • 1 Introduction: PEER provides a comprehensive benchmark covering 14 biologically relevant tasks across protein function, structure, localization, protein-protein interaction, and protein-ligand interaction prediction.Its splits are designed to test generalization in real-world settings, including out-of-distribution evaluation for protein engineering.
  • 1 Introduction: PEER evaluates traditional feature engineering, sequence encoders, and large-scale pre-trained protein language models on individual tasks and under multi-task learning.The benchmark examines both task-specific supervised adaptation and shared learning across tasks.
  • 1 Introduction: Pre-trained protein language models perform best on most individual tasks, while jointly training multiple tasks further improves performance.These findings are reported as the benchmark’s main experimental result.
  • 1 Introduction: PEER maintains public community leaderboards and plans to extend evaluation beyond sequence-based methods to structure-based approaches.The benchmark is intended as a starting point for continued community evaluation.

2 Related Work

Prior protein benchmarks generally target specific modeling areas, whereas PEER combines diverse protein understanding tasks and evaluates both single-task and multi-task sequence modeling.

  • 2 Related Work: Earlier protein representation methods extracted physicochemical or statistical features, while later sequence models adapted NLP techniques to large protein corpora.The related work describes transitions from engineered features to word2vec- and doc2vec-inspired approaches.
  • 2 Related Work: Existing benchmarks include CASP for structure prediction, CAFA for functional annotation, TAPE for sequence encoders, FLIP for fitness, TDA for drug discovery, and ATOM3D for 3D biomolecule understanding.These benchmarks differ in task coverage and modeling focus.
  • 2 Related Work: Table 1 organizes each benchmark task by category, dataset source, split sizes, and evaluation metric.The caption defines regression, classification, accuracy, and RMSE abbreviations.
  • 2 Related Work: PEER addresses the limited versatility of prior benchmarks by covering function, localization, structure, protein-protein interaction, and protein-ligand interaction prediction.It also evaluates multi-task learning in addition to conventional single-task learning.

3 Benchmark Tasks

PEER contains diverse protein sequence tasks spanning function, localization, structure, protein-protein interaction, and protein-ligand interaction prediction. Its task designs use biologically meaningful targets and splits intended to test generalization across mutations, homologs, and structural similarity.

  • 3 Benchmark Tasks: PEER includes 14 tasks grouped into five categories, representing proteins as amino-acid sequences and ligands as molecular graphs.The task table records categories, data sources, dataset statistics, and evaluation metrics.
  • 3.1 Protein Function Prediction: Function prediction tasks cover fluorescence, stability, β-lactamase activity, and solubility, using splits that test transfer across mutation orders or dissimilar sequences.Examples include training on mutants with three or fewer mutations and testing on mutants with four or more, or removing training sequences with at least 30% identity to test sequences.
  • 3.2 Protein Localization Prediction: Localization tasks predict either one of 10 cellular locations or the coarser membrane-bound versus soluble distinction, with splits testing generalization across homologous proteins.DeepLoc supplies the localization data and homologous sequences are clustered by sequence identity for splitting.
  • 3.3 Protein Structure Prediction: Structure tasks predict residue contacts, global folds, or local secondary structures, including remote-homology evaluation through held-out superfamilies.Contact prediction emphasizes medium- and long-range contacts, while fold classification uses 1,195 categorical fold labels.
  • 3.4 Protein-Protein Interaction Prediction: The benchmark also includes protein-protein interaction tasks and protein-ligand interaction prediction relevant to complex modeling, function understanding, and drug discovery.Protein-ligand interaction recognition is framed as a machine-learning task for small-molecule and target-protein interactions.

4 Methods

PEER evaluates feature engineering, sequence encoders, and pre-trained protein language models across diverse protein tasks using single- and multi-task learning pipelines.

  • Model Classes and Task Pipelines: PEER evaluates feature engineering, sequence encoders, and pre-trained protein language models across diverse protein understanding tasks.The benchmark covers function, localization, structure, protein-protein interaction, and protein-ligand interaction prediction.
  • Task-Specific Pipelines: Protein function, localization, and structure tasks use residue-, residue-pair-, or protein-level embeddings depending on the prediction target.An MLP predictor maps the selected representation to the task output.
  • Task-Specific Pipelines: Protein-protein interaction prediction uses siamese sequence encoders, while protein-ligand prediction combines protein sequence and ligand graph encoders.Both pipelines pass concatenated embeddings to an MLP predictor.
  • Single-Task Learning: Single-task learning optimizes a task-specific loss independently for each benchmark task.The model parameters are trained to minimize the loss for one selected task.
  • Multi-Task Learning: Multi-task learning pairs a center task with an auxiliary task to improve generalization under comparable training budgets.The setup accounts for substantially different training-set sizes across tasks.

5 Experiments

Experiments compare baseline model families in single-task and multi-task settings across the PEER benchmark. Pre-trained ESM-1b is strongest for most single tasks, while multi-task learning benefits CNN and ESM-1b more consistently than a scratch-trained Transformer.

  • Single-Task Results: DDE statistical features significantly outperform Moran physicochemical features on all available tasks.The result supports the informativeness of statistical characteristics in protein sequence segments.
  • Single-Task Results: ESM-1b achieves the best performance on 10 of 14 single-task benchmark tasks and remains effective both frozen and fine-tuned.The benchmark evaluates its use as a feature extractor and with joint parameter fine-tuning.
  • Multi-Task Results: Multi-task learning improves ESM-1b on all 13 applicable tasks when contact prediction is the auxiliary task.Fold classification and secondary structure prediction each improve 9 applicable tasks for ESM-1b.
  • Multi-Task Results: Multi-task learning improves shallow CNN on 9 of 13 applicable tasks with contact or secondary structure prediction as auxiliary tasks.Fold classification is less beneficial for CNN and produces 7 degraded results.
  • Multi-Task Results: Multi-task learning provides the least benefit to a scratch-trained Transformer and can produce degraded results depending on the auxiliary task.Contact prediction and secondary structure prediction lead to 9 and 8 degraded results, respectively, for Transformer.

6 Conclusions and Future Work

The paper presents PEER as a comprehensive benchmark comparing single- and multi-task protein sequence models. Its results identify shallow CNN as a low-cost competitive baseline, ESM-1b as a strong protein language model, and auxiliary-task selection as important for multi-task learning.

  • Conclusions: PEER compares three model types under single-task and multi-task learning for general protein sequence understanding.The benchmark evaluates feature-based models, sequence encoders, and pre-trained protein language models.
  • Conclusions: Shallow CNN is a competitive, low-computational-cost baseline, while ESM-1b is a superior protein language model in the benchmark.The conclusions summarize the single-task comparisons across PEER tasks.
  • Conclusions: Multi-task learning benefits CNN and ESM-1b, but effective auxiliary-task selection remains important.The conclusion identifies auxiliary-task selection as a key issue for future multi-task research.
  • Future Work: The current benchmark could be strengthened by adding more tasks, datasets, and baseline models.Planned extensions include Gene Ontology and Enzyme Commission tasks, plus ProtT5 and UniRep baselines.

7 Broader Societal Impacts

PEER evaluates protein sequence encoders across five types of protein-understanding tasks, supporting assessment across varied real-world applications. The benchmark also raises responsible-use concerns because validated models could facilitate harmful activities.

  • 7 Broader Societal Impacts: PEER evaluates sequence encoders across five protein-understanding task types to assess their general effectiveness in real-world applications.The task groups cover protein function, localization, structure, protein-protein interaction, and protein-ligand interaction prediction.
  • 7 Broader Societal Impacts: Validated models could also boost harmful activities such as designing harmful drugs, motivating future guidelines for responsible benchmark use.

B.1 Balanced Metrics on Classification Tasks

The benchmark uses weighted F1 to assess two highly imbalanced multi-class tasks while retaining accuracy in the main paper for comparability. The resulting baseline ranking is nearly unchanged.

  • B.1 Balanced Metrics on Classification Tasks: Weighted F1 accounts for severe class imbalance in fold classification and subcellular location prediction by averaging classwise F1 scores with class weights.Fold classification has classes with one sample versus others with tens or hundreds, while subcellular-location class sizes differ by more than 10-fold.
  • B.1 Balanced Metrics on Classification Tasks: The baseline ranking under weighted F1 is almost unchanged from accuracy: shallow CNN remains best from-scratch, and ESM-1b remains state of the art.
  • B.1 Balanced Metrics on Classification Tasks: Accuracy remains the main-paper metric for these tasks because conclusions are consistent and prior benchmark results commonly report accuracy.

C Ablation Studies

Ablations examine how sequence truncation affects localization and fold classification, with longer truncations improving performance across the evaluated models. Results are averaged over three runs with standard-deviation error bars.

  • C Ablation Studies: Figure 1 evaluates truncation length for localization and fold classification and training-set size for β-lactamase activity prediction.
  • C Ablation Studies: All Figure 1 results average three runs using seeds 0, 1, and 2, with standard deviations shown as error bars.
  • C Ablation Studies: Longer sequence truncation improves performance for DDE, CNN, and ESM-1b on both localization prediction and fold classification.Truncation is applied from the start of each protein sequence.

C.2 Effect of Training Set Size

Increasing the training set consistently improves β-lactamase activity prediction. Multi-task performance also depends on the auxiliary task and the tradeoff parameter.

  • C.2 Effect of Training Set Size: Performance of DDE, CNN, and ESM-1b monotonically increases as β-lactamase activity training samples grow from 100 to 4,158.The full training set contains 4,158 samples.
  • C.2 Effect of Training Set Size: The training-size results support collecting more labeled data for predicting protein fitness landscapes.
  • C.2 Effect of Training Set Size: For CNN multi-task learning, contact prediction as an auxiliary task benefits the center task from larger α, whereas secondary-structure prediction degrades under large α.With secondary structure as auxiliary, peak performance occurs for α between 0.1 and 1.0.

C.4 Training Center and Auxiliary Tasks under Single- and Multi-Task Learning

Under multi-task learning, the β-lactamase center task converges faster and generalizes better, while the auxiliary task trains less effectively. The auxiliary task can nevertheless reduce center-task variance across random seeds.

  • C.4 Training Center and Auxiliary Tasks under Single- and Multi-Task Learning: Multi-task learning accelerates convergence and improves validation generalization for the β-lactamase center task, while worsening auxiliary-task training.The β-lactamase task is designated as the center task; the auxiliary task is secondary structure prediction.
  • C.4 Training Center and Auxiliary Tasks under Single- and Multi-Task Learning: Figure 2 compares training losses and validation metrics for β-lactamase and secondary structure prediction under single- and multi-task learning, averaging three runs with standard-deviation error bars.The figure also examines how the tradeoff parameter α affects CNN multi-task learning.
  • C.4 Training Center and Auxiliary Tasks under Single- and Multi-Task Learning: Including secondary structure prediction as an auxiliary task largely reduces β-lactamase variance across random seeds.The paper suggests this may reflect stabilization from the auxiliary task's lower variance.
Loading 2206.02096v2…