PaperlayerView exact revision on arXiv

Source-linked AI summary

Protein Language Models and Structure Prediction: Connection and Progression

Bozhen Hu, Jun Xia, Jiangbin Zheng, Cheng Tan, Yufei Huang, Yongjie Xu, Stan Z. Li

arXiv:2211.16742v1q-bio.QMcs.AIcs.LG

TL;DR

Protein structure prediction from sequence remains challenging, and mainstream pipelines rely heavily on costly MSA searches. This survey synthesizes pLM and PSP advances, concluding that LMs and pLMs can aid biochemical, biomedical, and bioinformatic domains.

  • Problem

    Researchers lacked a systematic survey connecting pLMs with structure prediction and charting developments relevant to developing suitable protein algorithms.

    Source: §1 · p. 5

  • Method

    The paper systematically synthesizes pLM and PSP advances, covering their relationships, databases, structure-prediction methods, and pLM roles in protein folding.

    Source: §9 · p. 26

  • Results

    The authors conclude that LMs and pLMs can aid specific biochemical, biomedical, and bioinformatic domains.

    Source: §9 · p. 26

  • Takeaways & Limitations

    The survey is intended to help researchers develop suitable algorithms and tackle challenging protein problems.

    Source: §1 · p. 5

    Current pLM-based models still lack interpretability.

    Source: §7 · p. 23

Abstract

from arXiv · show

The prediction of protein structures from sequences is an important task for function prediction, drug design, and related biological processes understanding. Recent advances have proved the power of language models (LMs) in processing the protein sequence databases, which inherit the advantages of attention networks and capture useful information in learning representations for proteins. The past two years have witnessed remarkable success in tertiary protein structure prediction (PSP), including evolution-based and single-sequence-based PSP. It seems that instead of using energy-based models and sampling procedures, protein language model (pLM)-based pipelines have emerged as mainstream paradigms in PSP. Despite the fruitful progress, the PSP community needs a systematic and up-to-date survey to help bridge the gap between LMs in the natural language processing (NLP) and PSP domains and introduce their methodologies, advancements and practical applications. To this end, in this paper, we first introduce the similarities between protein and human languages that allow LMs extended to pLMs, and applied to protein databases. Then, we systematically review recent advances in LMs and pLMs from the perspectives of network architectures, pre-training strategies, applications, and commonly-used protein databases. Next, different types of methods for PSP are discussed, particularly how the pLM-based architectures function in the process of protein folding. Finally, we identify challenges faced by the PSP community and foresee promising research directions along with the advances of pLMs. This survey aims to be a hands-on guide for researchers to understand PSP methods, develop pLMs and tackle challenging problems in this field for practical purposes.

1 Backgrounds

Protein structure prediction (PSP) remains difficult because sequence changes can have unpredictable effects on protein shape, while experimental structures are scarce and physics-based simulation is computationally challenging. This survey connects language models (LMs) and protein LMs (pLMs), reviews their methods and PSP applications, and identifies challenges and future directions.

1 Backgrounds

  • PSP is difficult because small sequence changes can drastically alter shape, similar amino acids may have little effect, and flexible chains can fold into many shapes.Protein structures determine function and support biological processes, making PSP important across applications from therapeutics to materials.Source: §1 · p. 3
  • Only 190 thousand structures were in PDB, versus 528 million sequences in UniParc and 10 million annotated images in ImageNet; physics-based simulations also face computational and modeling limits.Experimental structure determination is expensive; computational PSP has traditionally pursued physical interactions or evolutionary principles, with sequence databases supporting MSA-based evolutionary methods.Source: §1 · p. 3
  • Self-supervised pLMs trained on millions to billions of proteins learn sequence representations for PSP and other tasks, and helped drive AlphaFold2’s success at CASP 14.Masked-language-modeling encoders learn by reconstructing corrupted residues; labeled proteins are scarce because experiments are time-consuming and resource-intensive.Source: §1 · pp. 3–4
  • The survey reviews LM and pLM architectures, pre-training, databases, applications, and PSP methods, including single-sequence prediction and predictions for complexes and other molecular partners.It also covers traditional physics- and machine-learning-based methods for structural-feature prediction and PSP.Source: §1 · p. 4
  • The authors present the work as the first survey connecting pLMs and structure prediction, compiling resources and identifying limitations, unsolved problems, and future research directions.The collected resources include LMs and pLMs, PSP methods, pre-training databases, and paper lists.Source: §1 · p. 5

2 Notions and Terms

The glossary defines protein structure at sequence, local, whole-protein, and subunit-arrangement levels, alongside concepts used to describe evolutionary relationships, structural representations, prediction tasks, and language models.

2 Notions and Terms

  • Protein structure ranges from an amino-acid sequence to local secondary elements, a protein’s 3D shape, and the arrangement of subunits in multisubunit proteins.Sequences are reported from the N-terminus to the C-terminus; secondary structure can use 3 states (H, E, C) or 8 finer-grained states.Source: §2 · p. 5
  • An MSA aligns three or more sequences and indicates homologous regions, while coevolution describes interdependent evolutionary changes and templates are homologous 3D structures.MSA assumes recognizable evolutionary homology; coevolution between protein residues is described by Ochoa and Pazos (2014).Source: §2 · pp. 5–6
  • A contact map represents residue contacts as a distance-thresholded binary matrix, torsion angles describe rotations around backbone bonds, and PSP predicts 3D structure from sequence.The two torsion angles are ϕ around N−Cα and ψ around Cα−C; a contact map is a reduced representation of protein structure.Source: §2 · p. 6
  • Other protein tasks include assigning biological or biochemical functions, predicting mutation effects on stability, designing proteins, and refining starting models toward native states.Gene Ontology classifies function annotations into molecular function, biological process, and cellular component.Source: §2 · p. 6
  • A language model is a probability distribution over words or sentences, and an embedding is a learned, low-dimensional continuous representation of discrete variables.The glossary also defines NLP as computer processing and understanding of text and spoken words.Source: §2 · p. 6

3 Protein and Language

Protein and human language share hierarchical, reusable structure and long-distance dependencies, motivating the use of NLP methods for protein data. However, proteins are not languages, and protein tokenization still lacks a well-defined, biologically meaningful method.

3 Protein and Language

  • Proteins and languages have hierarchical, reusable modular elements, while protein folding principles also resemble linguistic grammar (Ferruz and Höcker, 2022; Ofer et al., 2021a; Bryngelson et al., 1995; Leopold et al., 1992).The four levels of protein structure are compared by analogy to letters, words, sentences, and texts.Source: §3 · pp. 7–8
  • Long-range dependencies connect distant residues that are close in space, and MSA coevolution statistics can infer such residue contacts (Ochoa and Pazos, 2014; Vorberg, 2017; Zerihun and Schug, 2018).The passage compares these dependencies to grammatical agreement between distant words (Choshen and Abend, 2019).Source: §3 · p. 8, Figure 3 · p. 8
  • Protein tokenization remains comparatively undeveloped and lacks a well-defined, biologically meaningful algorithm, despite tokenization's role in language-model training.The passage identifies protein tokenization as a possible direction for unlocking the secrets of proteins.Source: §3 · p. 8

4 Language Models

The section traces language models from recurrent architectures to attention-based Transformers and surveys how pretraining objectives and later model developments shaped pretrained LMs. It explains the roles of RNNs, LSTMs, attention, and Transformer-based models in processing sequences and learning language representations.

4.1 Recurrent Neural Networks (RNNs) and Long Short-term Memory (LSTM)

  • RNN hidden states process variable-length sequences; LSTMs address vanishing gradients and learn long-term dependencies, while BiLSTMs use information from both directions.LSTM forget and input gates control cell-state retention and updates, and an output gate selects what the state outputs.Source: §4.1 · pp. 9–10

4.2 Attention Mechanism and Transformer

  • Transformers replace recurrence and convolutions with stacked self-attention, which weights encoder states and lets models process entire inputs at once.Attention helps address information loss from relying on an RNN or LSTM encoder’s final state; multi-head attention attends across representation subspaces, while positional encodings convey sequence order.Source: §4.2 · p. 10

4.3 Pre-trained Language Models

  • Pretraining followed by fine-tuning is framed as transfer learning, and Transformers are identified as the common backbone for pretrained LMs.The section says pretrained LMs achieved state-of-the-art results on almost all NLP tasks and points to Table 1 for typical general-domain models.Source: §4.3 · p. 11, Table 1 · p. 11

4.3.1 GPT and BERT

  • GPT predicts next tokens autoregressively, whereas BERT uses a bidirectional encoder trained with masked-language modelling and next-sentence prediction.The section presents GPT and BERT as landmarks that opened the era of large-scale, deeply pretrained LMs.Source: §4.3.1 · pp. 11–12

4.3.2 Post GPT and BERT Era

5 Protein Language Models

Protein sequences are well suited to language models, which can capture complex amino-acid dependencies. The section surveys LSTM and transformer pLMs, their training strategies, applications, and use in protein structure prediction.

5.1 LSTM Protein Language Models

  • LSTM pLMs learn transferable protein representations, while ProSE adds contact and structural-similarity supervision to improve protein-function prediction.UDSMProt and SeqVec use LSTM variants to capture long-range dependencies; CPCProt learns embeddings through an InfoNCE objective.Source: §5.1 · p. 13

5.2 Transformer Protein Language Models

  • PMLM (He et al., 2022) considered dependencies among masked tokens to capture inter-residue correlations (coevolution), and this modeling was demonstrated to improve performance on the TAPE contact benchmark.ESM-1b used a deep Transformer and masking to build context-rich sequence representations; another approach used fully differentiable adversarial masking.Source: §5.2 · pp. 13–14

5.2.1 Multi-source Knowledge Enhancement of Protein Representations

  • pLMs can integrate sequence with MSAs, function, structure, and biological priors, but auxiliary information and tasks must be selected carefully.MSA Transformer alternates row and column attention; ProteinBERT and OntoProtein incorporate GO, while PEER results indicate suitable auxiliary tasks can boost performance.Source: §5.2.1 · p. 14

5.2.2 Transformer Models Designed for Protein Structure Prediction

  • AF2’s Evoformer repeatedly updates MSA and pair representations through gated attention and triangle blocks, producing patterns for accurate structure prediction.Its 48 unshared-weight blocks use row- and column-wise attention, outer-product mean updates, triangle multiplicative updates, and triangle self-attention; PSP-trained pLMs can also support other structure and function tasks.Source: §5.2.2 · p. 14, Figure 6 · p. 15
  • ESM-2 scales ESM-1b to 15 billion parameters and achieves the lowest validation perplexity and highest CASP14 TM-score among smaller ESM models.The passage attributes improved PSP with little or no evolutionary information to increasing pLM size; AminoBERT maps single-sequence representations to backbone structures, while OmegaPLM and AntiBERTy target structural or antibody applications.Source: §5.2.2 · p. 15

5.2.3 Other Applications

  • pLMs also support property-conditioned sequence generation, mutation-effect prediction, protein design, inverse folding, and recognition of distant homologs.ProGen conditions generation on protein properties, MSA2Prot models sequence probabilities autoregressively, and ProtTucker uses contrastive learning for distant-homolog recognition.Source: §5.2.3 · pp. 15–16

6 Methods of Protein Structure Prediction (PSP)

The section traces PSP from structural-feature prediction and traditional sampling to feature-constrained and end-to-end folding, then highlights pLM pipelines that predict from single sequences and newer extensions.

6.1 Structural Features Prediction

  • Structural-feature prediction spans 1D attributes and 2D contacts or distances, with neural, recurrent, attention-based, and regression methods; real-value distance prediction is harder than classification (Xu, 2018).Although such tasks have limited practical value given AF2’s accurate structures, they remain useful for method comparison and studying sequence–structure–function relationships; DeepHomo targets inter-protein contacts and Geoformer addresses triangular inconsistency.Source: §6.1 · pp. 17–18

6.2 Traditional Methods for PSP

  • Traditional PSP methods generated structures through fragment assembly, Monte Carlo or other conformational searches, and energy-based or experimental restraints.Later approaches incorporated predicted contacts, secondary structure, or evolutionary information as folding restraints, including distance geometry and Rosetta-based sampling.Source: §6.2 · p. 18

6.3.1 Structural Features-Based Methods

  • Feature-based pipelines predict contacts or distances before modeling 3D structures, but their accuracy remains limited by the component steps.Methods predict inter-residue orientations as well as distances, then use energy minimization and full-atom relaxation; AlphaFold also generated structures directly through gradient descent, avoiding exhaustive sampling.Source: §6.3.1 · pp. 18–19, Figure 8 · p. 19

6.3.2 End-to-end PSP

  • AF2 achieved near-experimental accuracy in most cases and substantially outperformed competitors in CASP14, using an Evoformer and iterative structure module.Its structure module uses Invariant Point Attention and shared-weight layers to update single representations and backbone frames; RoseTTAFold instead exchanges information across sequence, distance, and coordinate tracks.Source: §6.3.2 · pp. 19–20, Figure 9 · p. 20

6.3.3 Single Sequence Structure Prediction via pLMs

  • Single-sequence pLM pipelines address the MSA-search bottleneck and enable accurate structure prediction for orphan proteins.OmegaFold combines OmegaPLM embeddings with Geoformer, whose stacked layers improve contact accuracy while reducing inconsistency; ESMFold, HelixFold-Single, RGN2, and trRosettaX-Single offer related approaches.Source: §6.3.3 · p. 21, Figure 10 · p. 21

6.3.4 Diversified Progress in the Protein Field

  • Later work extends PSP to complexes, nucleic acids, refinement, and generative structures, while AF2 segments may remain uncertain and GNNRefine performs relatively poorly on AF2 structures.Methods include pLM-assisted complex modeling, RoseTTAFoldNA for protein–nucleic acid structures, and SE(3)-equivariant all-atom refinement; diffusion models have generated diverse, high-quality structures.Source: §6.3.4 · pp. 21–22

7 Discussion: Limitations and Future Trends

The discussion argues that pLM research must address the cost and limitations of large models while developing protein-specific methods, more interpretable models, and reliable data and benchmarks. It also identifies unresolved structure-prediction and modeling tasks as opportunities for further progress.

Discussion: Limitations and Future Trends

  • Billion-parameter pLM training is costly and environmentally burdensome, so research should prioritize adapting pretrained models through transfer learning and distillation rather than training from scratch.DeepMind used 128 TPU v3 cores to train and fine-tune AF2 over one week; proposed distillation can transfer MSA, structure, and function information, as in NLP.Source: §7 · p. 22
  • AF2 may poorly predict mutation effects, structure-triggered tasks may not transfer, and larger models may not improve zero-shot fitness (Nijkamp et al., 2022).These limitations leave room for different models tailored to different problems.Source: §7 · p. 22
  • ESM-2 results suggest low-scale models saturate at high evolutionary depth but continue improving with size at low depth; suitable biological and physical priors may reduce model size and boost performance.The discussion also identifies evolutionary information, structural levels, and Gene Ontology information as possible additions.Source: §7 · pp. 22–23
  • Multi-task or multimodal learning over heterogeneous data merits further exploration, alongside robust structure prediction and understanding how sequence mutations affect protein function.The discussion connects mutation effects to genetic disease and a functional landscape in which function constrains sequence.Source: §7 · p. 23
  • Because current pLM architectures largely reuse NLP language models and masking strategies despite differences between proteins and language, protein-specific architectures and methods are needed.The discussion presents suitable protein tokenization as an appealing direction.Source: §7 · pp. 23–24
  • Limited interpretability obscures protein-folding mechanisms, and AF2-like models lack detailed molecular and chemical interactions needed for mechanistic studies and structure-based drug design.The discussion proposes visualization and combining protein tokenization with Bayesian modeling, Monte Carlo strategies, energy functions, Markov random fields, direct coupling analysis (Weigt et al., 2009), or diffusion models.Source: §7 · pp. 23–24
  • Protein databases may contain incorrect information, and newly built datasets require attention to leakage, bias, and ethics; complete, reliable, and fair benchmarks are essential for evaluating models.The discussion notes that groups sometimes build datasets to meet specific needs, including ProteinKG25 for protein sequence and function embeddings (Zhang et al., 2022b).Source: §7 · p. 24
  • Unsolved challenges include unified DNA, RNA, and protein engineering models, RNA–protein relationship interpretation, complex structure and conformation prediction, and PSP below 0.5Å resolution.The discussion says single-protein, protein–ligand, and RNA-complex structure prediction and conformation modeling are more complicated than PSP and remain unsolved.Source: §7 · p. 24

8 Databases

The section surveys protein sequence, structure, classification, and knowledge-graph databases, alongside CASP as a community-wide test of structure prediction methods.

8 Databases

  • UniProt provides reviewed and automatically annotated protein records, clustered UniRef sequences, and the non-redundant UniParc archive; PDB stores experimentally determined structures.As of October 2021, PDB had about 170k multi-chain structure files; PSP [Liu et al., 2022b] used 570k true-structure sequences and 745k complementary distillation sequences after AF2 appeared.Source: §8 · pp. 24–25
  • CATH classifies PDB-derived domains by class, architecture, topology, and homologous superfamily; CATH-Gene3D v4.3 includes 151 million domains in 5,841 superfamilies.Structured Transformer used a cluster-split CATH4.2 dataset with 18,025 training, 1,637 validation, and 1,911 test chains.Source: §8 · p. 25
  • AlphaFoldDB collects AlphaFold-predicted structures, with its latest release containing over 200 million entries for UniProt.The passage describes AlphaFold's structure prediction accuracy as competitive with experiments.Source: §8 · p. 25
  • BFD is a large metagenomic sequence-profile database; BFD30 contains 65 million clusters at 30% sequence identity, and searches are slower but more sensitive.BFD sequences were clustered with MMseqs2/Linclust at 30% identity and 90% coverage; cluster-derived MSAs can also be used.Source: §8 · p. 25
  • Pfam classifies protein families and domains using HMM-generated annotations and alignments; its recent version contains 19,632 families.The pfamseq profile HMM is queried against a UniProtKB-based sequence database to find homologues for a Pfam entry.Source: §8 · p. 25
  • ProteinKG25 unifies Gene Ontology terms and protein annotations in a knowledge graph for training language models with GO information.Its graph contains Protein-GO and GO-GO triplets, alongside protein and gene-term entities and relations.Source: §8 · p. 25
  • Uniclust clusters UniProtKB sequences at 90%, 50%, and 30% identity and adds Pfam, SCOP, and PDB matches that may be absent from UniProt annotations.Unlike UniRef's CD-HIT clustering, Uniclust uses MMseqs2, which the passage associates with higher consistency in functional annotation.Source: §8 · p. 25
  • SCOP and SCOPe classify protein structural domains by structural similarity and evolutionary relationships; SCOPe ASTRAL 2.08 includes over 50K classified sequences with known structures.SCOP2 was released in 2020, while SCOPe corrects SCOP errors and incorporates the ASTRAL database.Source: §8 · pp. 25–26

9 Conclusion

The paper surveys advances in protein language models (pLMs) and protein structure prediction (PSP), including their relationship and role in protein folding. It also introduces relevant databases, discusses limitations and future directions, and anticipates applications across biochemical, biomedical, and bioinformatic domains.

9 Conclusion

  • The survey covers pLMs, PSP methods, their relationship, and how pLMs contribute to protein folding.It also reviews background, protein representation learning, and related tasks.Source: §9 · p. 26
  • The paper introduces databases relevant to pLMs and PSP.Source: §9 · p. 26
  • The authors discuss limitations, possible ways to address them, and future trends, and expect LMs and pLMs to aid biochemical, biomedical, and bioinformatic domains.Source: §9 · p. 26

Related papers

Loading 2211.16742v1…