PaperlayerView exact revision on arXiv

Source-linked AI summary

Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformers

Hadi Abdine, Michail Chatzianastasis, Costas Bouyioukos, Michalis Vazirgiannis

arXiv:2307.14367v3q-bio.QMcs.CLcs.LG

TL;DR

Protein-function prediction commonly assigns predefined labels, which can oversimplify function. Prot2Text fuses graph-based structural and sequence representations to generate free-text descriptions; its RGCN encoder outperforms a vanilla-Transformer baseline on BLEU and BERT scores.

  • Problem

    Predefined-label classification can oversimplify protein functionality, motivating a free-text formulation of protein-function prediction.

    Source: §1 · p. 1

  • Method

    Prot2Text fuses GNN-derived protein structure graphs with ESM sequence representations and uses pretrained GPT-2 to generate free-text descriptions.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Over five absolute BLEU-score points and three BERT-score points, RGCN outperformed the vanilla-Transformer baseline.

    Source: §4 · p. 6

  • Takeaways & Limitations

    The released dataset provides 256,690 protein structures, sequences, and textual function descriptions for benchmarking and model comparison.

    Source: §1 · p. 2

    The RGCN encoder is not pretrained, unlike the ESM encoder, and may struggle to capture complex patterns in protein data.

    Source: §6 · p. 7

Abstract

from arXiv · show

In recent years, significant progress has been made in the field of protein function prediction with the development of various machine-learning approaches. However, most existing methods formulate the task as a multi-classification problem, i.e. assigning predefined labels to proteins. In this work, we propose a novel approach, Prot2Text, which predicts a protein's function in a free text style, moving beyond the conventional binary or categorical classifications. By combining Graph Neural Networks(GNNs) and Large Language Models(LLMs), in an encoder-decoder framework, our model effectively integrates diverse data types including protein sequence, structure, and textual annotation and description. This multimodal approach allows for a holistic representation of proteins' functions, enabling the generation of detailed and accurate functional descriptions. To evaluate our model, we extracted a multimodal protein dataset from SwissProt, and demonstrate empirically the effectiveness of Prot2Text. These results highlight the transformative impact of multimodal models, specifically the fusion of GNNs and LLMs, empowering researchers with powerful tools for more accurate function prediction of existing as well as first-to-see proteins.

1 Introduction

The introduction argues that label-based classification oversimplifies protein function and proposes generating free-text descriptions from protein sequence and structure. It presents Prot2Text, its claimed generation advantage from combining graph and sequence information, and a released multimodal dataset.

  • The dataset combines SwissProt-derived annotations with protein structures obtained from AlphaFold and is publicly released.Source: §1 · pp. 1–2
  • Prot2Text combines GNN and ESM protein representations in an encoder and uses pretrained GPT-2 to generate free-text protein descriptions.The introduction frames this as a shift from assigning predefined labels to describing protein function in text.Source: §1 · pp. 1–2
  • The authors report that integrating graph and sequence information leads to better protein-text generation capabilities.Source: §1 · p. 2
  • The released dataset contains 256,690 protein structures, sequences, and textual function descriptions for benchmarking and model comparison.Source: §1 · p. 2

2 Related Work

Related work covers transformer models, multimodal learning, GNNs, and protein representation methods. It notes that existing protein-text frameworks use sequence information but do not target detailed textual description generation.

Transformers

  • Transformer architectures have been extended from language tasks to protein sequences, images, and molecular SMILES representations.Source: §2 · p. 2

Multimodal models

  • Earlier protein-text frameworks ProtST Xu et al. (2023) and ProteinDT Liu et al. (2023) encode protein sequences but do not generate detailed textual descriptions.Source: §2 · p. 2
  • Multimodal precedents include CLIP for image-text representations and MolT5 for translation between molecule representations and natural language.CLIP combines vision and language encoders; MolT5 supports molecule captioning and generation.Source: §2 · p. 2

Graph Neural Networks

  • GNNs propagate and refine node features through iterative information exchange, and heterogeneous variants handle graphs with different node and edge types.Source: §2 · p. 2

Protein Representation Learning

  • Protein representations include sequence-only, structure-based graph, and hybrid approaches, with Deep-FRI combining pretrained language-model sequence features and protein structures.Source: §2 · p. 3

3 Methodology

Prot2Text encodes protein structure as a graph and protein sequence with ESM, fuses the representations, and uses a GPT-2 decoder to generate function descriptions. Its pipeline combines graph construction and encoding, sequence projection, multimodal fusion, and causal text generation.

3 Methodology

  • Protein graphs contain amino-acid nodes and typed edges representing sequence adjacency, spatial proximity, and hydrogen-bond interactions.Node features include local structural and physicochemical properties.Source: §3 · p. 3
  • An RGCN updates node representations using relation-specific neighborhoods, then mean-pools the final-layer nodes into a graph vector.The resulting vector is used in subsequent text generation.Source: §3 · p. 3
  • ESM2-35M encodes the protein sequence, and a trainable projection maps its amino-acid representations into the graph embedding dimension.This produces the sequence representation used in fusion.Source: §3 · pp. 3–4
  • Four fusion blocks combine each amino-acid representation with the graph representation, enriching sequence vectors with graph context.The fusion uses element-wise addition followed by projection and normalization.Source: §3 · p. 4
  • A GPT-2-initialized decoder uses cross-attention to condition description generation on the protein representation and is trained with causal language modeling.Descriptions are limited to 256 tokens.Source: §3 · p. 5

4 Experimental Results

Prot2TextBASE leads the evaluated encoder models across all reported text-generation metrics, while larger model sizes generally improve performance at increased inference cost. Results also indicate that the best size depends on test–train sequence similarity.

Dataset

  • The dataset contains 256,690 SwissProt proteins with sequences, AlphaFold structures, and textual descriptions; the train, validation, and test splits contain 248,315, 4,172, and 4,203 proteins.CD-HIT limits maximum similarity between sets to 40%.Source: §4 · p. 5

Baselines

  • Comparisons include unimodal RGCN, ESM, and scratch-trained vanilla-Transformer encoders, concatenated RGCN+ESM, and RGCN × vanilla-Transformer, which replaces pretrained ESM2 with a vanilla-Transformer trained from scratch.The vanilla-Transformer baseline matches the configuration and parameter count of pretrained ESM2-35M.Source: §4 · p. 5

Training and Metrics

  • Models were trained for 25 epochs, and generation quality was evaluated using BLEU, Rouge-1/2/L, and BERT Score.Each Base model took approximately 12 hours to train.Source: §4 · pp. 5–6

Results

  • Prot2TextBASE achieves the highest reported scores: BLEU 35.11, Rouge-1 50.59, Rouge-2 42.71, Rouge-L 48.49, and BERT Score 84.3.All models use the same GPT-2 decoder.Source: §4 · p. 6, Table 1 · p. 7
  • Concatenating graph representations with ESM output yields slightly worse results than ESM alone, unlike Prot2Text’s fusion approach.The authors note that indiscriminately increasing parameter count could degrade performance.Source: §4 · p. 6

Scaling to Larger Models

  • Larger Prot2Text models outperform smaller ones on most metrics, but Prot2TextMEDIUM (398M parameters) balances performance and inference cost.Prot2TextMEDIUM performs best for test proteins at 20–30% identity or with no training counterpart; Prot2TextLARGE performs better at higher similarity.Source: Figure 2 · p. 5, §4 · p. 6, Table 2 · p. 7

5 Conclusion

Prot2Text generates free-text protein-function descriptions from graph and sequence inputs, and the paper concludes that this offers a more nuanced alternative to predefined-label classification. The authors also release a multimodal dataset for benchmarking this task.

Conclusion

  • Prot2Text predicts protein function in free text from graph and sequential inputs, as an alternative to classification with predefined labels.The conclusion describes its generated descriptions as detailed and coherent.Source: §5 · p. 7

6 Limitations and Future Work

The paper identifies the lack of pretraining for its RGCN encoder as a limitation and proposes graph-specific pretraining as future work.

Limitations and Future Work

  • The RGCN encoder is not pretrained, unlike ESM, and may therefore struggle to capture complex patterns in protein data.The authors propose exploring auxiliary graph-related pretraining tasks using graph- or node-level information.Source: §6 · p. 7

A.1 Tokenization

Prot2Text truncates descriptions at 256 tokens and adapts the GPT-2 tokenizer with separate start- and end-of-sequence tokens.

A.1 Tokenization

  • Figure 4 shows the distribution of description token counts per sample and highlights the 256-token threshold in red.Source: Figure 4 · p. 7
  • 256 tokens covers 98.7% of descriptions, so Prot2Text truncates descriptions to a maximum length of 256 tokens.The dataset contains 256,690 proteins with unique amino-acid sequences and 48,251 unique function descriptions.Source: §A.1 · pp. 7–8
  • The Prot2Text tokenizer adds separate start-of-sequence and end-of-sequence tokens to the GPT-2 tokenizer.The added tokens receive corresponding embeddings while the rest of the GPT-2 word-embedding matrix is kept intact.Source: §A.1 · p. 8

A.2 Text Generation

Text generation uses the Transformers library, with validation outputs generated during training and configurable decoding settings for examples.

A.2 Text Generation

  • Validation descriptions are generated every 500 training steps with greedy search, one beam, and a maximum length of 256 tokens.Source: §A.2 · p. 8
  • 37.09 was the best validation BLEU score for Prot2TextBASE, reached at step 23,000 after scores began stabilizing beyond epoch 20.Figure 5 tracks validation BLEU across training iterations; higher is better.Source: §A.2 · p. 8
  • The Figure 3 examples use Prot2TextBASE with length penalty 2.0, no-repeat ngram size 3, and early stopping enabled.The general generation defaults are length penalty 1.0, no-repeat ngram size 0, and early stopping disabled.Source: §A.2 · p. 8

A.3 CO2 Emission Related to Experiments

The experiments used 23,000 computation hours on Tesla V100-SXM2-32GB hardware and produced an estimated 393.3 kgCO2eq in emissions.

A.3 CO2 Emission Related to Experiments

  • 393.3 kgCO2eq is the estimated total emissions from the experiments, with 0 percent directly offset.Source: §A.3 · p. 8
  • The experiments used 23,000 computation hours on Tesla V100-SXM2-32GB hardware with a 300W TDP.Source: §A.3 · p. 8
  • The reported carbon efficiency of the private infrastructure was 0.057 kgCO2eq/kWh.Emissions were estimated using the MachineLearning Impact calculator presented in Lacoste et al. (2019).Source: §A.3 · p. 8
  • Figure 5 tracks Prot2TextBASE validation BLEU across training iterations, with higher scores indicating better performance.Source: Figure 5 · p. 9
Loading 2307.14367v3…