OntoProtein: Protein Pretraining With Gene Ontology Embedding

1 INTRODUCTION

• Existing protein language models learn powerful representations but generally do not sufficiently capture biological factual knowledge, which is sparse and varied in sequence data.Knowledge graphs offer structured biology facts, but combining them with protein sequences raises knowledge-encoding and heterogeneous-information-fusion challenges.Source: §1 · pp. 1–2
• OntoProtein integrates external knowledge graphs into protein pretraining using contrastive learning with knowledge-aware negative sampling to jointly optimize knowledge-graph and protein-sequence embeddings.Its knowledge-embedding objective encodes GO annotation descriptions, while masked language modeling follows existing protein pretraining approaches.Source: §1 · p. 2
• OntoProtein is model-agnostic and can be plugged into protein tasks without additional inference overhead because it adds training objectives without modifying model architecture.The approach is intended to retain protein language models’ masked-language-modeling ability while adding knowledge-graph supervision.Source: §1 · p. 2
• ProteinKG25 contains about 612,483 entities and 4,990,097 triples with GO-aligned node descriptions, and provides inductive and transductive data splits.The authors describe it as the first large-scale knowledge-graph dataset intended to facilitate protein pretraining.Source: §1 · p. 2
• Experiments across TAPE, protein-protein interaction, and protein function prediction demonstrate OntoProtein’s effectiveness, with the authors reporting promising improvements across protein tasks.The introduction summarizes extensive experiments but gives no task-specific result values.Source: §1 · pp. 2–3

2 METHODOLOGIES

• OntoProtein jointly optimizes knowledge embedding and masked protein modeling to incorporate Gene Ontology knowledge into protein representations.The overall objective combines the two training objectives, with a hyperparameter weighting the knowledge-embedding term.Source: §2 · p. 3, §2.5 · p. 5

2.1 HYBRID ENCODER

• Protein-sequence and GO-description encoders produce representations in a shared space, while relation embeddings are randomly initialized and trained.The protein representation is mean-pooled and projected through an affine transformation; GO token embeddings are averaged.Source: §2.1 · pp. 3–4

2.2 KNOWLEDGE EMBEDDING

• The knowledge graph represents GO-to-GO and protein-to-GO facts as triples, encoding proteins from sequences and GO entities from annotation text.The hybrid encoders place these heterogeneous descriptions into a common semantic space.Source: §2.2 · p. 4

2.3 MASKED PROTEIN MODELING

• Masked protein modeling masks each amino-acid token with 15% probability and trains the model to predict the masked tokens.The objective is jointly trained with knowledge embedding, and the approach can use other pre-trained models.Source: §2.3 · p. 4

2.4 CONTRASTIVE LEARNING WITH KNOWLEDGE-AWARE NEGATIVE SAMPLING

• Knowledge-aware negative sampling uses GO knowledge to generate negatives; GO-to-GO triples replace entities within the same ontology aspect, while protein-to-GO triples replace only the tail.The three aspects are molecular function, cellular component, and biological process.Source: §2.4 · pp. 4–5

2.5 PRE-TRAINING OBJECTIVE

• The pre-training objective combines the knowledge-embedding loss with the masked-protein-modeling loss, weighted by a hyperparameter.The paper states that the approach can be embedded into existing fine-tuning scenarios.Source: §2.5 · p. 5

3 EXPERIMENT

• The experiments evaluate OntoProtein on TAPE, protein-protein interaction, and protein function prediction.The function-prediction evaluation uses transductive and inductive settings, while the TAPE evaluation includes secondary-structure and contact tasks.Source: §3 · p. 5, §3.1 · p. 6

3.1 DATASETS

• ProteinKG25 combines GO terms and relations with annotated proteins and their sequences, forming a unified graph with 4,990,097 triples.GO terms are aligned to names and descriptions; proteins are aligned to Swiss-Prot sequences.Source: §3.1 · p. 6

3.2 RESULTS

• OntoProtein improves all token-level TAPE tests, beats TAPE Transformer and ProtBert on secondary structure and contact prediction, and is comparable to MSA Transformer.The authors note that OntoProtein does not use multiple sequence alignments and performs less well on protein engineering, homology, and stability regression tasks.Source: Table 1 · p. 6, TAPE BENCHMARK · p. 7
• OntoProtein outperforms PIPR and improves on GNN-PPI variants in the smaller SHS2K dataset, while remaining comparable to GNN-PPI variants on larger datasets.The comparison includes three PPI datasets of different sizes.Source: PROTEIN-PROTEIN INTERACTION · p. 7, Table 2 · p. 7
• OntoProtein improves BPO prediction by 4% in the transductive setting and 2% in the inductive setting, with comparable performance on MFO and CCO.The authors note a severe long-tail issue and suggest that knowledge injection may weaken tail representations.Source: Table 3 · p. 7, PROTEIN FUNCTION PREDICTION · p. 8

3.3 ANALYSIS

• Across short- through long-range contact settings and three P@L metrics, OntoProtein exceeds other methods; Figure 4 compares one protein’s attention head with its contact labels.The analysis tests sequence lengths from 6 ≤ seq < 12 to 24 ≤ seq.Source: §3.3 · p. 8, Table 4 · p. 8, Figure 4 · p. 8

3.4 DISCUSSION

• The authors report that gains over large-corpus pre-trained models remain relatively small and that ProteinKG25 covers only a small subset of proteins.They identify adding new Gene Ontology facts and investigating when and how to inject external knowledge as future work.Source: §3.4 · pp. 8–9

4.1 PRE-TRAINED LANGUAGE MODELS

• Protein language-model studies explored scaling, pretraining choices, and biologically relevant representations, but did not explicitly incorporate external knowledge.Source: §4.1 · p. 9

4.2 KNOWLEDGE-ENHANCED LANGUAGE MODELS

• NLP knowledge-enhanced models inject relational facts into language models, and KEPLER jointly optimizes knowledge embeddings and language representations (Wang et al. (2021b)).Source: §4.2 · p. 9
• OntoProtein extends knowledge-enhanced pretraining to proteins by integrating external gene ontology knowledge into protein language models.Source: §4.2 · p. 9

• OntoProtein integrates gene ontology knowledge into protein language models, with reported positive results on widespread protein tasks and compatibility with existing pretrained parameters.Users can adopt available pretrained parameters without modifying the architecture.Source: §5 · p. 9
• Future work could improve OntoProtein by selecting more informative gene ontology knowledge and extend the approach to protein-design sequence generation.Source: §5 · p. 10

A.1 CONSTRUCTION OF PROTEINKG25

• The dataset uses GO annotations and protein sequences as node descriptions and provides transductive and inductive evaluation settings.Source: §A.1 · p. 15
• ProteinKG25 contains 4,990,097 triples, 612,483 entities, and 31 relations, combining protein-GO annotations with GO-GO relations.It includes 565,254 proteins and 47,229 GO terms.Source: §A.1 · p. 15
• Protein-GO annotations are relatively sparse for leaf terms; the authors suggest annotation complexity or few proteins associated with specific terms as possible explanations.GO-term depth is measured by the shortest path to the root.Source: §A.1 · p. 16
• To address the long-tailed Protein-GO relation distribution, preprocessing adds fine-grained relations for frequently occurring GO terms in MF, CC, and BP.The selected terms are the top 10 in MF and CC and top 20 in BP.Source: §A.1 · p. 16

A.2 DOWNSTREAM TASK DEFINITION

• Secondary-structure prediction labels each amino acid as helix, strand, or other, while contact prediction labels amino-acid pairs as contacting or not.Source: §A.2 · p. 16
• Remote homology detection classifies sequences into 1,195 protein folds, while fluorescence and stability landscape prediction are sequence-level regression tasks.Source: §A.2 · p. 16
• Protein-protein interface prediction maps a pair of protein sequences to a binary label.Source: §A.2 · p. 16
• Protein function prediction is framed as sequence-level classification or as predicting a protein-to-Gene-Ontology link.Source: §A.2 · p. 16

A.3 EXPERIMENTAL SETTINGS

• OntoProtein pre-training uses the BERT training protocol, including its optimizer and learning-rate schedule.The setup follows Elnaggar et al. (2020).Source: §A.3 · p. 16
• The experiments use PyTorch and Nvidia V100 GPUs.Source: §A.3 · p. 16
• Pre-training sets γ to 12.0 and uses 128 negative samples in Equation 1.Source: §A.3 · p. 16

A.4 DATAFLOW OF ONTOPROTEIN

• OntoProtein jointly trains on batches of (protein-seq, protein-go, go-go).Figure 7 depicts the dataflow.Source: §A.4 · p. 16, Figure 7 · p. 18
• Figure 5 contrasts the initial and pre-processed relation distributions.Source: Figure 5 · p. 17
• Figure 6 presents three Gene Annotation datasets from different periods, chosen to generate pre-training and protein-function evaluation sets.Source: Figure 6 · p. 17
• Table 5 lists GO-term entity categories and their specific relation types.Source: Table 5 · p. 18
• Table 6 presents the downstream-task hyperparameters.Source: Table 6 · p. 18
• GO annotations associate a gene or gene product with a GO term and are identified by the gene product, term, reference, and evidence.GO terms also have description text, called the GO statement.Source: Table 7 · p. 18