Source-linked AI summary
ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, Burkhard Rost
TL;DR
ProtTrans addressed gaps in protein sequence annotation and the costs and limits of evolutionary information by exploring protein language-model pretraining and transfer learning. The resulting embeddings captured constraints relevant to protein structure and function, while downstream models appeared competitive across several prediction tasks and self-supervised transfer learning opened new perspectives on existing problems.
Problem
Protein sequences vastly outnumber experimentally determined structures, while evolutionary information is computationally expensive, unavailable for some proteins, and may blur differences between family members.
Method
ProtTrans explored scaling protein language models and databases, compared autoregressive with auto-encoding pretraining, and transferred extracted embeddings into subsequently trained supervised models.
Results
Embeddings from pre-trained protein language models captured constraints relevant for protein structure and function, and small CNN/FNN models using embeddings alone appeared competitive for secondary structure, localization, and membrane classification.
Takeaways & Limitations
Self-supervised pretraining combined with transfer learning tapped unlabeled sequence data and opened new perspectives and solutions for existing protein-prediction problems.
Takeaways & Limitations
Open questions remain about auxiliary tasks, more efficient transformer training, data preprocessing and sampling, and tailoring supervised pipelines to particular applications.
Abstract
from arXiv · showhide
Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models taken from NLP. These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The LMs were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw protein LM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks. The first was a per-residue prediction of protein secondary structure (3-state accuracy Q3=81%-87%); the second were per-protein predictions of protein sub-cellular localization (ten-state accuracy: Q10=81%) and membrane vs. water-soluble (2-state accuracy Q2=91%). For the per-residue predictions the transfer of the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without using evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that protein LMs learned some of the grammar of the language of life. To facilitate future work, we released our models at https://github.com/agemagician/ProtTrans.
1 INTRODUCTION
ProtTrans applies self-supervised language modeling to protein sequences to learn transferable representations, motivated by gaps between sequence, structure, and annotation and by the costs and limits of evolutionary-information methods. The project evaluates model scaling and pre-training strategies against existing evolutionary-information-based solutions.
- Language-model approach: Self-supervised language models learn from sequential order rather than requiring annotations, then expose learned information as embeddings for transfer learning.Supervised models can use these embeddings while expensive pre-training is performed on high-performance computing infrastructure.
- Motivation: Protein sequences encode structure and function, but experimentally determined structures are far fewer than known sequences.This creates a sequence-structure gap and motivates prediction methods for closing the broader sequence-annotation gap.
- Limitations of existing methods: Evolutionary-information methods require computationally expensive database searches and may be unavailable for intrinsically disordered or dark-proteome proteins.They can also average over protein families and may not distinguish different proteins within the same family.
- Protein language modeling: Protein language models treat sequences as sentences and amino acids as words, using structural and functional constraints as an analogy to linguistic grammar.These models extract features directly from individual protein sequences.
- Project objectives: ProtTrans investigates how scaling protein language-model training and database size, and how autoregressive versus auto-encoding pre-training, affect supervised transfer performance.The models are compared with existing state-of-the-art solutions that use evolutionary information.
2 METHODS
The methods train protein language models on clustered sequence databases, extract token- or protein-level embeddings, and use them in downstream experiments. The study spans multiple architectures, sequence-length strategies, datasets, and large-scale training configurations.
- Data: UniRef50, UniRef100, and BFD provide clustered protein-sequence corpora, with BFD combining UniProt and metagenomic sequences after duplicate removal.BFD was about eight times larger than the largest datasets previously used for protein language models.
- Embedding transfer: The study extracts last-hidden-state embeddings and transfers them to supervised models without fine-tuning the pre-trained language models.Embeddings support both per-residue and per-protein prediction tasks.
- Prediction levels: Token-level embeddings can feed residue predictors, while pooling across sequence length produces fixed-size protein representations for protein-level classifiers.The overview describes CNN-based residue prediction and FNN-based protein classification.
- Protein language models: Six NLP language-model architectures were trained on protein sequences: T5, Electra, BERT, Albert, Transformer-XL, and XLNet.The models include auto-encoding and autoregressive approaches with architecture-specific reconstruction or attention mechanisms.
- Sequence length: Positional-information strategies determine supported sequence lengths: several models use staged fixed positional encodings, whereas Transformer-XL reuses hidden-state memory across fragments.T5 learns relative positional encodings shared across layers and attention heads.
- Large-scale training: Training configurations vary across UniRef and BFD datasets and use Summit, TPU Pods, distributed frameworks, and model-specific optimization settings.Table 2 reports configurations for ProtTXL, ProtBert, ProtXLNet, ProtAlbert, ProtElectra, and ProtT5.
3 RESULTS
ProtTrans embeddings captured biophysical, structural, organismal, and functional information from unlabeled protein sequences and supported strong downstream predictions. Performance generally improved with larger pre-training sample counts, while ProtT5 variants approached or exceeded MSA-based methods without evolutionary alignments.
- Unsupervised protein LMs informative: ProtTrans embeddings captured biophysical amino-acid features and clustered proteins by structural class, organism, and putative function.ProtT5-XL-U50 and ProtBert-BFD produced especially separable protein-level representations.
- Per-residue secondary structure prediction: All tested protein LMs improved significantly over context-free feature extractors, while most embedding-only approaches remained below NetSurfP-2.0.ProtT5-XL-U50 was the exception, reaching nearly identical performance without multiple sequence alignments.
- Training trends: Performance correlated with pre-training samples (Spearman’s ρ=0.62), and comparing ProtT5-XL with ProtT5-XXL suggested sample exposure could matter more than model size.The authors noted that model capacity and training duration may trade off.
- Per-protein localization & membrane prediction: Mean-pooling outperformed min-, max-, and concatenated pooling for fixed-length protein representations used in localization and membrane classification.Min/max pooling performed almost fourteen percentage points worse for localization and about three points worse for membrane/other classification.
- Per-protein localization & membrane prediction: ProtT5 models fine-tuned on UniRef50 outperformed DeepLoc for ten-state localization without MSAs, whereas membrane classification remained 1-2 percentage points below DeepLoc.For membrane/other prediction, fine-tuning did not increase performance.
4 DISCUSSION
The discussion links downstream performance to pre-training scale, bidirectional context, and the information captured by protein-LM embeddings. These embeddings approached or matched evolutionary-information methods while reducing dependence on costly sequence searches, although important scope questions remain.
- Training protein LMs longer most important: Training on more samples correlated with downstream performance, whereas model-size effects were less consistent.The reported association was Spearman’s ρ=0.62; the authors suggest a trade-off between model capacity and sample throughput.
- Protein LMs top without MSAs: Protein-LM embeddings encoded rudimentary structural and functional information before supervised training, including attention to residues involved in zinc binding.The visual analyses used t-SNE, UMAP, and attention inspection to identify these signals, while warning that high-dimensional projections can be misinterpreted.
- Protein LMs top without MSAs: ProtT5-XL embeddings eliminated the performance gap with evolutionary-information methods across the evaluated tasks, sometimes outperforming MSA-based state of the art.Other language models generally remained below evolutionary-information methods, while newer contextual models improved over earlier LM-based approaches.
- Protein LMs top without MSAs: Combining evolutionary information with embeddings may yield the most accurate methods, but would sacrifice single-sequence prediction advantages.The stated advantages include large-scale analysis and tasks such as single amino acid variant effect prediction.
- Protein LMs top without MSAs: Bidirectional context substantially improved protein-LM performance compared with the unidirectional ProtTXL model.The authors contrast ProtXLNet with ProtTXL and note that combining independently derived forward and backward representations may partially compensate.
- Open questions: The authors identify open questions about auxiliary objectives, efficient transformer training, data processing, supervised-pipeline optimization, and combining evolutionary information with AI.They treated secondary-structure and localization prediction primarily as proxies for evaluating protein language models rather than independent endpoints.
- Overall discussion: HPC-enabled training and supervised prediction scaled to the largest data sets used in the field, demonstrating state-of-the-art-level performance without multiple sequence alignments.The conclusion attributes this demonstration to combining high-performance-computing solutions with protein language models and supervised prediction methods.
5 CONCLUSION
The conclusion reports that protein-LM embeddings capture constraints relevant to protein structure and function and can support competitive predictions without MSAs. Self-supervised pre-training and transfer learning therefore make use of unlabeled protein sequences for new prediction approaches.
- Embedding-based prediction: Protein-LM embeddings captured constraints relevant to protein structure and function.The conclusion reports this for embeddings extracted from the final LM layers.
- Embedding-based prediction: Small supervised models using embeddings achieved competitive predictions for secondary structure, localization, and membrane-versus-other classification without MSAs.The conclusion states that these methods challenged state-of-the-art MSA-based methods while using embeddings as exclusive input.
- Self-supervision and transfer learning: Self-supervised pre-training combined with transfer learning exploited unlabeled protein data for new perspectives on existing problems.The conclusion presents this combination as opening new perspectives and solutions.
SUPPLEMENTARY ONLINE MATERIAL (SOM)
The supplementary material examines protein-language-model corpora, embedding content, attention patterns, supervised architectures, prediction performance, and inference costs. Together, these analyses characterize both what the models encode and how they support downstream protein prediction.
- Language-model corpora: UniRef50, UniRef100, and BFD are compared by sequence count, residue count, storage size, tensor-converted size, and amino-acid distribution.The comparisons cover datasets used for protein language-model pre-training.
- Unsupervised embedding analysis: t-SNE analyses qualitatively assess whether embeddings capture amino-acid properties, protein structure, function, cellular localization, membrane association, and lineage.The analyses use SCOPe, experimentally annotated enzyme classes, and redundancy-reduced localization datasets.
- Unsupervised embedding analysis: ProtT5 produces better clustering than ProtBERT across the analyzed biological levels when visual entropy is interpreted as a proxy for learned information.The comparison spans biophysical and biochemical properties, protein functions, cellular compartments, and kingdoms of life.
- Attention analysis: Attention visualization identifies residue-pair weighting and shows a ProtAlbert head attending mainly to zinc-coordinating residues in a zinc-finger protein.The analysis interprets attention scores as potentially informative about structural motifs and model behavior.
- Supervised prediction: CNN, FNN, LSTM, and logistic-regression classifiers are compared for secondary-structure prediction, with CNN reported as best overall and more computationally efficient than LSTM.The comparison uses ProtBERT-BFD embeddings and evaluates three- and eight-state prediction on CASP12 and NEW364.
- Supervised prediction: ProtT5-XL-U50 outperforms NetSurfP-2.0 for 57% of NEW364 proteins using single sequences, while MSA generation is compared with protein-LM embedding generation.The reported comparison covers 208 of 364 proteins and uses MMseqs2 settings aligned with NetSurfP-2.0.
- Protein-level representations: Mean pooling outperforms min- and max-pooling for fixed-length protein representations, while concatenating all three pooling outputs matches mean pooling for membrane classification but performs worse for localization.These representations are evaluated with an FNN for ten-state localization and membrane-bound versus water-soluble classification.
B Membrane vs Soluble
The supplied material lists protein categories associated with the membrane-versus-soluble analysis, including structural classes, functional classes, and size-related categories.
- Structural classes: The categories include all-alpha, all-beta, and alpha-and-beta structural classes.
- Protein categories: The listed protein categories include multi-domain proteins, membrane or cell-surface proteins, and small proteins.
- Functional classes: The functional categories include oxidoreductases, transferases, hydrolases, lyases, isomerases, ligases, and translocases.
E Lineage: Kingdoms
The supplied material lists feature categories used alongside lineage analysis, spanning amino-acid properties, molecular size, and subcellular localization.
- Amino-acid properties: Amino-acid property categories include aromatic and aliphatic hydrophobicity, positive and negative charge, and polar neutrality.
- Molecular size: Size categories distinguish small, medium, and big molecules, with small defined as below 130 Dalton and big as above 150 Dalton.
- Localization: Subcellular localization categories include nucleus, cytoplasm, mitochondrion, extracellular space, and cell membrane.
- Localization: Additional localization categories include endoplasmic reticulum, plastid, Golgi apparatus, lysosome or vacuole, and peroxisome.
E Lineage: Kingdoms
The supplied material repeats category labels spanning structural classes, protein groupings, enzyme functions, amino-acid properties, molecular size, and cellular localization.
- Amino-acid properties: Amino-acid labels distinguish aromatic and aliphatic hydrophobicity, positive and negative charge, and polar neutrality.
- Molecular size: The size labels are small below 130 Dalton, medium, and big above 150 Dalton.
- Localization: Localization labels include nucleus, cytoplasm, mitochondrion, extracellular space, cell membrane, endoplasmic reticulum, plastid, Golgi apparatus, lysosome or vacuole, and peroxisome.
E Lineage: Kingdoms
The t-SNE analysis assessed whether ProtBert embeddings captured biochemical properties and protein-level distinctions across localization, structure, and function.
- Embedding analysis: t-SNE projections assessed features extracted by ProtBert from unlabeled protein sequences.The analysis examined embedding structure rather than supervised labels.
- Amino-acid properties: The amino-acid analysis grouped residues by hydrophobic, charge, and polarity-related properties.The displayed categories include aromatic and aliphatic hydrophobic residues, positive and negative residues, and polar-neutral residues.
- Protein-level features: Protein-level analyses covered membrane status, cellular localization, structural class, and enzymatic function.Displayed categories include membrane and cell-surface proteins, cellular compartments, SCOPe structural classes, and enzyme classes.
E Lineage: Kingdoms
The t-SNE analysis assessed whether ProtAlbert embeddings represented amino-acid chemistry and several protein-level biological attributes.
- Embedding analysis: t-SNE projections assessed features extracted by ProtAlbert from unlabeled protein sequences.The figure tested whether the embeddings captured biophysical and biochemical information.
- Amino-acid properties: The amino-acid categories included aromatic and aliphatic hydrophobicity, charge, polarity, and size.Size categories were small (<130 Dalton), medium, and big (>150 Dalton).
- Protein-level features: Protein-level categories included cellular compartments, structural classes, protein types, and enzyme functions.The displayed labels span localization compartments, SCOPe classes, membrane or small proteins, and enzyme classes.
E Lineage: Kingdoms
The t-SNE analysis examined whether ProtXLNet embeddings captured amino-acid properties and multiple protein-level biological classifications.
- Embedding analysis: t-SNE projections assessed features extracted by ProtXLNet from unlabeled protein sequences.The figure evaluated biophysical and biochemical properties alongside protein-level distinctions.
- Amino-acid properties: The amino-acid categories represented hydrophobicity, charge, polarity, and molecular size.The labels distinguish aromatic versus aliphatic hydrophobicity and small, medium, versus big residues.
- Protein-level features: Protein-level categories covered cellular compartments, structural classes, protein types, and enzyme functions.The displayed labels include localization, SCOPe structure, membrane or small proteins, and enzyme classes.
E Lineage: Kingdoms
ProtTXL embeddings were evaluated with t-SNE across amino-acid chemistry, localization, membrane status, structural class, function, and lineage.
- Projection method: t-SNE projected mean-pooled last-layer ProtTXL representations into two dimensions for protein-level analysis.The representations were contextualized, fixed-size embeddings averaged over each protein sequence.
- Datasets: The analyses used redundancy-reduced DeepLoc data at 30% and SCOPe data at 40%.DeepLoc supported membrane-status and cellular-compartment analyses, while SCOPe supported structural, functional, and lineage-specific analyses.
- Comparison: ProtTXL formed the least dense clusters among the compared language models but captured some protein-function aspects.Transferases were identified as an example captured by ProtTXL but not by the other models in the comparison.
- Caveat: DeepLoc panels B and C are not directly comparable because proteins did not all have annotations for both features.The caption explicitly limits comparison between membrane or water-soluble status and cellular-compartment annotations.