PaperlayerView exact revision on arXiv

Source-linked AI summary

ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts

Minghao Xu, Xinyu Yuan, Santiago Miret, Jian Tang

arXiv:2301.12040v2q-bio.BMcs.LG

TL;DR

Sequence-only PLMs capture co-evolutionary information but do not explicitly acquire protein functions and other properties. The paper builds ProtDescribe and introduces ProtST, which combines unimodal and multimodal pre-training to enrich PLMs with biomedical-text information. ProtST-induced PLMs perform effectively across supervised representation learning, zero-shot classification, and text-to-protein retrieval tasks.

  • Problem

    Existing PLMs learn mainly from sequences, capturing co-evolutionary information without explicitly acquiring protein functions and other properties.

  • Method

    ProtST builds the ProtDescribe paired dataset and uses unimodal mask prediction, multimodal representation alignment, and multimodal mask prediction to enrich PLMs with textual property information.

  • Results

    ProtST-induced PLMs are generally effective across supervised learning, zero-shot protein classification, and zero-shot text-to-protein retrieval, outperforming vanilla PLMs on 21–24 of 24 benchmark metrics depending on the backbone.

  • Takeaways & Limitations

    ProtST supports protein representation learning and protein–text applications that include zero-shot prediction and functional retrieval without function annotations.

  • Takeaways & Limitations

    ProtDescribe has limited coverage of protein sequences and textual property descriptions.

Abstract

from arXiv · show

Current protein language models (PLMs) learn protein representations mainly based on their sequences, thereby well capturing co-evolutionary information, but they are unable to explicitly acquire protein functions, which is the end goal of protein representation learning. Fortunately, for many proteins, their textual property descriptions are available, where their various functions are also described. Motivated by this fact, we first build the ProtDescribe dataset to augment protein sequences with text descriptions of their functions and other important properties. Based on this dataset, we propose the ProtST framework to enhance Protein Sequence pre-training and understanding by biomedical Texts. During pre-training, we design three types of tasks, i.e., unimodal mask prediction, multimodal representation alignment and multimodal mask prediction, to enhance a PLM with protein property information with different granularities and, at the same time, preserve the PLM's original representation power. On downstream tasks, ProtST enables both supervised learning and zero-shot prediction. We verify the superiority of ProtST-induced PLMs over previous ones on diverse representation learning benchmarks. Under the zero-shot setting, we show the effectiveness of ProtST on zero-shot protein classification, and ProtST also enables functional protein retrieval from a large-scale database without any function annotation.

1. Introduction

Existing protein language models capture co-evolutionary information from sequences but do not explicitly acquire protein functions and other properties. ProtST addresses this gap by pairing protein sequences with biomedical property descriptions and using multimodal pre-training.

  • Sequence-only PLMs capture co-evolutionary information but cannot explicitly acquire protein functions or properties such as subcellular locations.
  • Textual property descriptions are intended to support generalization from known protein properties to unknown ones through semantic correlations.
  • ProtDescribe pairs protein sequences with biomedical descriptions covering names, functions, subcellular locations, and protein families.
  • ProtST combines a PLM, biomedical language model, and fusion module through unimodal mask prediction, multimodal alignment, and multimodal mask prediction.
  • ProtST-induced PLMs are evaluated on representation learning, zero-shot protein classification, and zero-shot text-to-protein retrieval.

2. Preliminaries

The paper formulates pre-training as learning protein representations from paired amino-acid sequences and biomedical text descriptions. These representations are intended to support supervised downstream prediction and zero-shot prediction.

  • A protein is represented as an amino-acid sequence S and a biomedical text description T.
  • The pre-training objective is to extract effective protein representations by using information from both sequences and descriptions.
  • PLMs commonly use masked protein modeling, predicting masked residues from sequence context to learn protein representations.
  • ProtST uses PubMedBERT to represent biomedical protein descriptions because biomedical language differs from general-domain text.

3. Method

ProtST combines sequence and biomedical-text modeling to inject protein properties into PLMs while preserving sequence-based information. Its aligned representations support supervised learning, zero-shot classification, and text-to-protein retrieval.

  • Framework overview: ProtST pre-trains protein sequences and biomedical texts jointly, then applies the resulting models to supervised and zero-shot applications.
  • Unimodal Mask Prediction: Unimodal mask prediction preserves co-evolutionary information by predicting masked residues from protein-sequence context.
  • Multimodal Representation Alignment: Multimodal representation alignment uses contrastive learning to increase similarity between corresponding protein sequences and text descriptions while reducing similarity between negative pairs.
  • Multimodal Mask Prediction: Multimodal mask prediction recovers corrupted sequences or descriptions using both modalities to model fine-grained residue–word dependencies.
  • Advantages over Self-Supervised PLMs: ProtST combines whole-protein property learning through representation alignment with residue-level property learning through multimodal mask prediction.
  • Advantages over Previous Works: Text-based property modeling is designed to generalize beyond fixed known properties by exploiting semantic correlations between known and unknown descriptions.

4. Experiments

Experiments evaluate ProtST on diverse supervised and zero-shot protein tasks using aligned sequence–text data and several pretrained backbones. ProtST generally improves benchmark performance, data efficiency, ensemble performance, and text-to-protein retrieval.

  • Experimental setup: ProtDescribe contains 553,052 aligned protein sequence–property-description pairs from Swiss-Prot annotations.The descriptions cover protein names, functions, subcellular locations, and protein families.
  • Experimental setup: ProtST enhances ProtBert, ESM-1b, and ESM-2 through multimodal pre-training with biomedical language representations.The evaluated variants are ProtST-ProtBert, ProtST-ESM-1b, and ProtST-ESM-2.
  • Experimental setup: The evaluation covers 11 benchmarks spanning localization prediction, fitness landscape prediction, and protein function annotation.Baselines include sequence encoders, vanilla PLMs, and OntoProtein.
  • Supervised benchmark results: ProtST-ProtBert, ProtST-ESM-1b, and ProtST-ESM-2 outperform their vanilla counterparts on 21, 22, and 24 of 24 benchmark metrics, respectively.ProtST-ESM-2 improves over vanilla ESM-2 on all 24 metrics.
  • Supervised benchmark results: ProtST-ProtBert surpasses OntoProtein on 22 of 24 benchmark metrics under a fair comparison.Both models start from ProtBert, enabling the comparison.
  • Supervised benchmark results: ProtST-ESM-1b performs best on fitness prediction, while ProtST-ESM-2 performs best on localization prediction and function annotation.ProtST-ESM-1b is best on 4 of 6 fitness metrics, and ProtST-ESM-2 is best on 7 of 8 function-annotation metrics.
  • Zero-shot classification: Zero-shot ProtST-ESM-1b matches or exceeds several few-shot classifiers on localization and reaction classification.It matches 3-shot supervised ProtST-ESM-1b and 5-shot supervised ESM-1b on localization, and exceeds two 7-shot nonparametric classifiers.
  • Zero-shot applications: Zero-shot ProtST-ESM-1b enhances all evaluated few-shot and full-shot baselines through ensemble-based decision-boundary refinement.The aligned representation space also supports retrieval of functional proteins without function annotations; among the top four heme-binding candidates, three have GO heme-binding annotations.

5. Related Work

Prior protein representation methods primarily use sequences, structures, families, or knowledge graphs, while text-supervised multimodal learning has been less studied for proteins. ProtST addresses this gap by incorporating biomedical text descriptions into protein sequence representations.

  • Protein representation learning: Sequence-based methods model proteins at evolutionary scale or within individual protein families, while structure-based methods represent proteins through their structures.These approaches form major existing directions in protein representation learning.
  • Multimodal representation learning: Text-supervised multimodal representation learning is established for images, video, speech, and molecules but has been less studied for proteins.OntoProtein instead learns protein representations using a knowledge graph.

6. Conclusions and Future Work

ProtST combines aligned protein sequences and biomedical property descriptions to support supervised learning, zero-shot classification, and text-to-protein retrieval. Future work targets broader data coverage, protein structures, and text-guided protein design.

  • Conclusions: ProtST builds ProtDescribe and injects protein-property information at different granularities into protein language models.The resulting models are evaluated on supervised learning, zero-shot protein classification, and zero-shot text-to-protein retrieval.
  • Future work: The current ProtDescribe dataset has limited coverage of protein sequences and textual property descriptions.The authors propose extracting information from massive PubMed biomedical articles to expand coverage.
  • Future work: Future work includes adding protein structures and pursuing text-guided controllable protein design.These directions extend the framework beyond text-to-protein retrieval.

A. Model Architecture for Pre-training

ProtST fuses protein-sequence and text representations through self- and cross-attention, then applies alignment and masked-token prediction heads to the fused outputs.

  • Fusion Module: The fusion module updates residue and word representations by attending within each modality and across the protein sequence and text description.Queries, keys, and values are derived separately for sequences and text before multi-head self- and cross-attention updates both modalities.
  • Fusion Module: Each fusion layer combines sequence self-attention with sequence-to-text attention, and text self-attention with text-to-sequence attention.The sequence update sums MHA(QS, KS, VS) and MHA(QS, KT, VT); the text update sums MHA(QT, KT, VT) and MHA(QT, KS, VS).
  • Fusion Module: The implementation uses one fusion layer with eight attention heads to limit fusion capacity and preserve PLM representation power.Multimodal mask prediction is performed on the fused residue and word representations.
  • Pre-training Heads: A two-layer MLP projects sequence and text representations for global contrastive alignment.The projections use ReLU nonlinearities and feed the global contrastive loss.
  • Pre-training Heads: Two-layer MLP heads predict masked residues from PLM representations and masked residues and words from fused multimodal representations.The multimodal head separately predicts masked protein tokens and masked text tokens.

B.1. More Pre-training Setups

The supplementary setups specify ProtST data construction, downstream prediction heads, evaluation metrics, prompting, and zero-shot classification procedures, with a reported fitness-prediction gain over ESM-1b.

  • Data Curation: ProtDescribe descriptions concatenate Swiss-Prot annotations for protein name, function, subcellular location, and protein family in that order.Field prefixes identify the annotation sources, including “PROTEIN NAME,” “FUNCTION,” “SUBCELLULAR LOCATION,” and “SIMILARITY.”
  • Training Configurations: ProtST training uses Adam for 20 epochs on four Tesla V100 GPUs, with a 1.0×10−5 learning rate for other modules and 1.0×10−6 for the PLM.Sequences longer than 450 residues are truncated, and PubMed-BERT remains frozen.
  • Downstream Prediction: Two-layer MLP heads produce classification logits, regression scores, or per-function logits for localization, fitness, and function annotation tasks.The task-specific outputs follow TorchDrug defaults.
  • Evaluation Metrics: Protein function annotation is evaluated with AUPR and Fmax, which summarize protein-function-pair precision and the maximum protein-centric F-score over thresholds.Fmax averages precision and recall over proteins before taking the maximum F-measure.
  • Prompt Engineering: For zero-shot localization, the pre-training prompt template outperforms name-only and natural-language templates, while location names outperform location descriptions.The selected format is “SUBCELLULAR LOCATION: {label}”.
  • Prompt Engineering: For reaction classification, the pre-training template performs best, and appending detailed scientific comments provides no further zero-shot benefit.The selected representation uses enzyme names and alternative names with the “FUNCTION: {Name} {AlterNames}” template.
  • Zero-shot Classification: The nonparametric few-shot classifier derives test logits from representation similarity to training proteins and converts them to probabilities with softmax.ProtST-ESM-1b and ESM-1b versions serve as zero-shot classifier baselines.
  • Fitness Prediction: 15.1% relative improvement over ESM-1b is reported for ProtST-ESM-1b on ProteinGym fitness prediction.The comparison identifies ProtST-ESM-1b as the best-performing model under the stated evaluation.

C.2. Comparisons with Alignment-based Methods

Alignment-based methods outperform ProtST-ESM-1b on ProteinGym substitution fitness prediction, but their ensemble with ProtST-ESM-1b performs best.

  • Comparison: Alignment-based EVE and GEMME outperform ProtST-ESM-1b because they additionally use homologous information from sequence alignments.The comparison reports UniProt-level Mean Spearman’s ρ.
  • Comparison: The ensemble “ProtST-ESM-1b + GEMME” outperforms both alignment-based methods, indicating complementary knowledge between the two approaches.The authors identify combining these method families as a promising direction for future work.

D. More Zero-shot Text-to-Protein Retrieval Results

ProtST-ESM-1b retrieves ligand-binding proteins from text prompts, while ablations show that all three pre-training losses contribute across benchmark metrics.

  • Ligand Retrieval: Text-to-protein retrieval visualizes top-4 candidates for ligand prompts alongside docking results, binding affinities, and GO molecular-function labels.The studies cover ATP, GTP, P5P, and NAD+ binder retrieval.
  • Ligand Retrieval: ProtST-ESM-1b retrieves binders for diverse ligands, including sparsely annotated ATP, GTP, P5P, and NAD+ binders.The reported GO annotation rates are 3.99% for ATP, 1.18% for GTP, 0.17% for P5P, and 0.05% for NAD+.
  • Ablation Study: Removing unimodal mask prediction, global alignment, or multimodal mask prediction reduces performance on 16, 20, or 19 of 24 benchmark metrics, respectively.The ablation results support retaining all three pre-training losses.

E.2. Ablation Study of Biomedical Language Model

The ablation compares PubMedBERT-abs and PubMedBERT-full as biomedical language models within ProtST-ESM-1b. PubMedBERT-full helps localization prediction, but PubMedBERT-abs performs better on most fitness and function-annotation metrics, consistent with ProtDescribe descriptions resembling abstracts.

  • The experiment evaluates both biomedical language models across localization, fitness, and function-annotation benchmark tasks.
  • PubMedBERT-full outperforms PubMedBERT-abs on all four localization-prediction metrics.
  • PubMedBERT-abs outperforms PubMedBERT-full on 10 of 12 fitness-prediction metrics and 5 of 8 function-annotation metrics.
  • ProtDescribe descriptions more closely resemble abstracts than full-text articles, helping explain why PubMedBERT-full is not superior overall.

F. More Visualization

ProtST-ESM-1b representations exhibit organized patterns for amino-acid biophysical properties and protein-level cellular localization. These visualizations indicate that the representations encode distinctions relevant to biochemical and biological properties, while retrieval figures examine zero-shot recognition of binding proteins.

  • Biophysical Properties of Amino Acids: t-SNE projections show distinct clusters for hydrophobic versus polar residues, including aliphatic versus aromatic amino acids.
  • Biophysical Properties of Amino Acids: The amino-acid clustering is coherent with respect to charge and size.
  • Biological and Biochemical Properties of Proteins: Protein representations show clustering patterns across cellular locations in subcellular and binary localization datasets.
  • Figure 10 evaluates zero-shot text-to-protein retrieval for ATP, GTP, P5P, and NAD+ binders using ProtST-ESM-1b.
Loading 2301.12040v2…