PaperlayerView exact revision on arXiv

Source-linked AI summary

Controllable Protein Design with Language Models

Noelia Ferruz, Birte Höcker

arXiv:2201.07338v2q-bio.BM

TL;DR

Designing proteins as proficiently as natural ones remains beyond contemporary research despite its potential value. This review surveys NLP and pretrained Transformer approaches for protein design and outlines six prospective applications.

  • Problem

    Despite great advances, contemporary research remains far from designing proteins as proficiently as those generated naturally.

    Source: Introduction · p. 1

  • Method

    The paper surveys NLP methods and pretrained language models for adaptation to protein research and design.

    Source: NLP has had an impact on protein… · p. 5, Conclusion · p. 12

  • Results

    The review outlines six prospective applications, ranging from generating sequences in unobserved protein-space regions to understanding sequence–structure–function relationships.

    Source: Conclusion · p. 12

  • Takeaways & Limitations

    The authors conclude that current NLP methods have great potential for adaptation to protein research.

    Source: NLP has had an impact on protein… · p. 5

    Applying NLP to proteins is challenged by less discernible token boundaries, while structural tokenization requires tertiary structures and more computation than word tokenization.

    Source: The language of proteins · p. 3

Abstract

from arXiv · show

The 21st century is presenting humankind with unprecedented environmental and medical challenges. The ability to design novel proteins tailored for specific purposes could transform our ability to respond timely to these issues. Recent advances in the field of artificial intelligence are now setting the stage to make this goal achievable. Protein sequences are inherently similar to natural languages: Amino acids arrange in a multitude of combinations to form structures that carry function, the same way as letters form words and sentences that carry meaning. Therefore, it is not surprising that throughout the history of Natural Language Processing (NLP), many of its techniques have been applied to protein research problems. In the last few years, we have witnessed revolutionary breakthroughs in the field of NLP. The implementation of Transformer pre-trained models has enabled text generation with human-like capabilities, including texts with specific properties such as style or subject. Motivated by its considerable success in NLP tasks, we expect dedicated Transformers to dominate custom protein sequence generation in the near future. Finetuning pre-trained models on protein families will enable the extension of their repertoires with novel sequences that could be highly divergent but still potentially functional. The combination of control tags such as cellular compartment or function will further enable the controllable design of novel protein functions. Moreover, recent model interpretability methods will allow us to open the 'black box' and thus enhance our understanding of folding principles. While early initiatives show the enormous potential of generative language models to design functional sequences, the field is still in its infancy. We believe that protein language models are a promising and largely unexplored field and discuss their foreseeable impact on protein design.

Introduction

Protein design aims to create tailored sequences that adopt structures with desired functions, but its combinatorial complexity makes de novo design time-consuming and current methods remain far from natural proficiency. Because protein sequences encode structure and function, NLP offers a promising avenue for design.

Introduction

  • Tailored proteins could help address environmental and medical challenges, yet contemporary research remains far from designing proteins as proficient as naturally generated ones [1].Examples include enzymes that metabolize plastic waste or hydrolyze pollutants, and vaccines designed promptly during a pandemic.Source: Introduction · p. 1
  • Protein design seeks sequences that adopt desired folds, but combinatorial complexity makes de novo design time-consuming and most functional proteins are instead optimized from natural scaffolds.Design algorithms search for low-energy sequences across a sequence–structure landscape with many possible rotamers and combinations.Source: Introduction · p. 1
  • Because amino acid sequences encode protein structure and function, sequence-based NLP methods offer an unexplored avenue for protein research.This approach uses sequence properties without biophysical constraints.Source: Introduction · p. 1

The language of proteins

Protein sequences and human languages share hierarchical organization and several analogous behaviors, making language models a promising perspective for protein design. But protein tokenization, interpretation, and evolution differ in ways that challenge applying NLP.

The language of proteins

  • Amino acids form structural elements and fragments that assemble into functional protein structures, paralleling letters forming words and words forming meaningful sentences (Fig. 1a).Protein domains can further assemble into larger quaternary complexes, analogous to sentences building text.Source: The language of proteins · pp. 1–2, Figure 1 · p. 2
  • Language and protein evolution share parallels, including common ancestry and reusable primordial peptides [5–7]; mutations, permutations, and grammatical but meaningless constructs have protein analogues [8–10].The passage compares language ancestry to LUCA and describes primordial peptides reused in different protein contexts.Source: The language of proteins · pp. 2–3, Figure 1 · p. 2
  • Protein token boundaries are less clear, generated sequences remain difficult to interpret and require experimental tests, and protein evolution reflects randomness and environmental pressure.Tokenization based on structural elements or conserved fragments would rely on tertiary structures and more computation than word tokenization.Source: The language of proteins · p. 3
  • Despite these challenges, an NLP-based viewpoint could enable greater control over protein design and encourage collaboration between artificial intelligence and biology.The authors present Transformers’ generative capabilities as a potential route toward novel functions and non-natural enzymatic reactions.Source: The language of proteins · p. 3

NLP has had an impact on protein research for decades

Protein research has repeatedly adapted methods developed for NLP, from shallow machine learning to neural networks and Transformers. Recent pretrained language models offer potential for protein research and design.

NLP has had an impact on protein research for decades

  • For decades, protein research applied NLP’s HMMs and SVMs to classification and labeling, including fold recognition, sequence classification, and cell localization; these methods remain state of the art for sequence homology detection.Figure 2 summarizes the parallels between methods in the two fields.Source: NLP has had an impact on protein… · p. 3, Figure 2 · p. 5
  • CNNs extended from NLP tasks to protein disorder, DNA-binding-site, and fold-classification prediction, but could not model the long-range information needed for global text comprehension or protein contacts.CNNs were used for NLP name-entity recognition, part-of-speech tagging, and semantic role labeling [17], and for protein predictions [18], [19], [20].Source: NLP has had an impact on protein… · p. 4
  • Inspired by RNNs’ capacity to learn long-term dependencies, Alley et al. developed the protein language model UniRep, which predicted sequence stability more accurately than previous methods [24].RNNs, particularly LSTMs, were used to create language models [23].Source: NLP has had an impact on protein… · p. 4
  • Attention mechanisms influenced protein research breakthroughs such as AlphaFold [25], while the Transformer improved most NLP tasks at lower computational cost [26].The Transformer began a new era of NLP, followed by adaptations including GPT, GPT2, and GPT3, which generated human-like, long, coherent articles [27], [28], [29].Source: NLP has had an impact on protein… · p. 4, Figure 2 · p. 5
  • Recent NLP developments have great potential for adaptation to protein research and could transform and dominate protein design in the years to come.The authors present this as potential, not an established outcome.Source: NLP has had an impact on protein… · p. 5

Attention mechanism and Transformers

Attention lets sequence models focus on relevant input parts, addressing the limitations of sequential processing; the Transformer builds on attention with parallel layers. Decoder-only GPT and encoder-only BERT illustrate distinct pretrained language-model designs.

Attention mechanism and Transformers

  • Attention lets seq2seq decoders inspect the whole input and focus on relevant parts, addressing RNNs’ performance and training-time degradation as sequences lengthen11.For example, a translation decoder can focus on ‘home’ when generating ‘maison’.Source: Attention mechanism and Transformers · p. 5
  • The 2017 Transformer applied attention throughout its architecture, using six parallel encoder and decoder layers to improve performance and enable faster training26.Encoder layers combine self-attention and feed-forward processing; decoder layers add encoder-decoder attention. Figure 3 contrasts this architecture with seq2seq and GPT/BERT variants.Source: Attention mechanism and Transformers · p. 5, Figure 3 · p. 7
  • Transformer attention embeds input tokens with positional information, computes Query, Key, and Value vectors, and uses scaled scores to weight values across eight parallel heads.The weighted values are summed into an output vector representing contributions from other words in the sentence.Source: Attention mechanism and Transformers · p. 6
  • GPT uses decoder-only autoregressive pretraining on unlabelled text followed by task-specific fine-tuning, improving state of the art on 9 of 12 tasks27.GPT-2 expanded parameters and training data tenfold over GPT, while GPT-3 had 100 times GPT-2’s parameters and performed well zero-shot on tasks such as code writing28,29.Source: Attention mechanism and Transformers · p. 7, Figure 3 · p. 7
  • BERT uses only the Transformer encoder and is pretrained to reconstruct corrupted input tokens, primarily producing text representations for downstream tasks such as classification31,43.Although autoencoders can also be used for text generation42, their common use is representation learning.Source: Attention mechanism and Transformers · pp. 7–8, Figure 3 · p. 7

Protein sequences are ideal candidates for Transformers

Protein databases—especially sequence databases—are growing rapidly, creating a vast, increasingly unlabeled protein space that Transformers may be able to exploit as they have large web corpora.

Protein sequences are ideal candidates for Transformers

  • Over 40 zettabytes of digital data accumulated in 2020, with estimates exceeding 80 zettabytes by 2025; growing corpora help Transformers build larger, more powerful models.The passage links the success of recent Transformers partly to ever-growing training corpora.Source: Protein sequences are ideal candidates… · p. 8
  • Protein-sequence characterization is growing faster than structural characterization, widening the unlabeled-to-labeled sequence-structure gap despite AlphaFold25,46-enabled structure growth and persistent functional-annotation delays.The authors describe this gap as increasing exponentially.Source: Protein sequences are ideal candidates… · p. 8, Figure 4 · p. 8
  • UniParc contained 441,169,278 protein-sequence entries in release 2021_03, providing a vast data pool that Transformers could exploit like unlabeled web corpora.The authors suggest this could stimulate a similar revolution in protein research.Source: Protein sequences are ideal candidates… · p. 8

Transformers for protein design

Protein language models have progressed from n-gram and LSTM representations to large pre-trained Transformers that support downstream protein predictions. Although publicly available pre-trained models for protein sequence generation were not yet available, the paper argues that GPT-like Transformers could enable controllable protein design.

Transformers for protein design

  • Before protein Transformers, n-gram and LSTM language models generated protein representations, and UniRep representations supported state-of-the-art predictions of sequence stability or function [24, 47].Yu et al. applied n-gram modelling, while Alley et al. developed the LSTM-based UniRep model.Source: Transformers for protein design · p. 8
  • ESM-1b, trained on 250 million protein sequences, produced representations that encode protein organization and supported predictions of mutational effects and long-range contacts [48].Its 33-layer encoder Transformer used the BERT architecture and training objective.Source: Transformers for protein design · pp. 8–9
  • ProtTrans adapted six Transformer architectures using over 390 billion amino acids and showed that protein embeddings could predict per-residue secondary structure and sub-cellular localization [49].The adapted architectures were Transformer-XL, BERT, Albert, XLnet, T5, and Electra.Source: Transformers for protein design · p. 9
  • No publicly available pre-trained models for protein sequence generation existed, though Gibbs sampling coupled to ESM-1b enabled direct sequence generation [50].The authors suggest GPT-2- or GPT-3-like Transformers could generate higher-quality protein sequences, extrapolating from their text-generation capabilities [42].Source: Transformers for protein design · p. 9
  • A GPT-like Transformer trained on protein sequences could generate new sequences while providing representations for downstream predictions of properties such as mutation effects.Fine-tuning on a protein family or fold could guide generation toward that group and extend its repertoire with sequences predicted to fold and have similar activities.Source: Transformers for protein design · p. 9

Tailored protein design

Conditional Transformers such as ProGen illustrate how tags can guide protein-sequence generation, while offering a way to explore property-controlled and novel protein designs.

Tailored protein design

  • CTRL’s conditional tags inspired ProGen, trained on 281 million protein sequences with over 1,100 UniParKB keyword terms, including biological process, cellular component, and molecular function [37, 53].ProGen showed high-quality-model-like perplexities, including on unseen protein families; generated sequences scored better than random sequences by Rosetta evaluation, and kinase completions stayed near the native protein’s Rosetta score. The top 100 protein G-binding variants by perplexity had better fitness scores than random mutations.Source: Tailored protein design · p. 9
  • After fine-tuning on five protein families, ProGen-generated lysozymes showed natural-range enzymatic activity, and X-ray characterization found that one variant recapitulated the native structure [54].Source: Tailored protein design · pp. 9–10
  • Conditional tags could enable generation of novel sequences while providing control over protein properties, extending Transformer-based protein language models beyond unconditioned generation.Source: Tailored protein design · p. 10
  • The authors envision tags for known properties, discovery of proteins in unexplored sequence-space regions for recurring properties, and novel functions from combinations such as hydrolase plus PET binding.Other examples include ATP binding, all-beta structure, membrane binding, and combining membrane binding with protease function.Source: Tailored protein design · p. 10

Enzyme, receptor, and biosensor design

Transformer models can conditionally generate molecules from proteins, suggesting that reversing the mapping could support receptor and biosensor design; reaction-to-protein generation could offer a route to designing enzymes for nonnatural catalysis.

Enzyme, receptor, and biosensor design

  • IBM RXN’s Molecular Transformer predicts reaction products from reactants and reagents, and its reversed form determines reactants for a target molecule through retrosynthesis [56, 57].The related protein-to-ligand model instead encodes protein sequences and generates potentially compatible ligand SMILES [58].Source: Enzyme, receptor, and biosensor design · pp. 10–11
  • Reversing protein-to-ligand generation could produce receptor sequences that recognize and bind specific ligands, advancing receptor and biosensor design [58].Grechishnikova’s model generates potentially compatible ligand SMILES from protein sequences; the proposed reversal would generate sequences from input SMILES.Source: Enzyme, receptor, and biosensor design · p. 11
  • Encoding chemical reactions as input and producing protein sequences could enable enzyme design for catalyzing reactions not found in nature [59].The authors suggest this approach could support biological strategies to reverse environmental pollution.Source: Enzyme, receptor, and biosensor design · p. 11

Explainable protein design

Explainable AI could help researchers understand deep learning models and reveal principles linking protein sequences to structures. Transformer attention offers a way to visualize amino-acid relationships, though explainability for protein design remains in its infancy.

Explainable protein design

  • XAI could clarify why deep learning models reach particular answers and help researchers develop rational protein designs with funnel-shaped folding energy landscapes.XAI has also helped drug discovery identify ligand pharmacophores associated with molecular activity61.Source: Explainable protein design · p. 11
  • Unlike traditionally explainable NLP methods such as HMMs and SVMs, deep learning models are black boxes addressed by five XAI approaches: feature importance, surrogate models, example-driven, provenance-based, and declarative induction62.Source: Explainable protein design · p. 11
  • Transformer attention scores can be visualized as saliency heatmaps, and adapting exBERT63 to protein models could expose amino-acid relationships and predicted properties.Such interfaces could also search protein fragments against training-corpus matches to illuminate relationships between proteins and support understanding of folding and design, although the field remains nascent.Source: Explainable protein design · p. 11

Is the future of protein design in the hand of big companies?

Training large Transformer models is resource-intensive and dominated by big companies, raising access and environmental concerns. The paper argues that public release, academic collaboration, and more efficient models could still broaden benefits to research.

Is the future of protein design in the hand of big companies?

  • 3.4-month doubling in compute used by the largest AI training runs compounds cost, access, and carbon-footprint concerns as models grow [64-67].GPT-3's 175 B parameters were estimated to require $12 million and over 10,000 days of GPU time.Source: Is the future of protein design in the… · pp. 11–12
  • All seven protein-based Transformer models published to date were led by or included large companies, but this need not create an imbalance.Source: Is the future of protein design in the… · p. 12
  • Publicly released large models can serve diverse downstream tasks, and company-academic collaborations may create opportunities for academia, though public release is not guaranteed.The paper cites AlphaFold and ProtTrans as examples of efforts involving public release.Source: Is the future of protein design in the… · p. 12
  • DistilBERT retains 97% of BERT performance at 40% smaller size, while Switch achieves up to 7x faster pre-training on the same compute as T5.The authors compare these efficiency efforts with molecular dynamics, which became more accessible through in-house GPU clusters and algorithmic solutions.Source: Is the future of protein design in the… · p. 12

Conclusion

The conclusion envisions six applications of NLP-derived Transformer methods to protein research, spanning sequence generation through model interpretability, and argues that protein language models could transform protein design and help address societal challenges.

Conclusion

  • Transformer models’ human-like text generation has prompted misuse concerns [41], but their capabilities and the analogy between language and protein sequences suggest benefits for protein research [71].Source: Conclusion · p. 12
  • Ordered by present applicability, the six envisioned uses include generating sequences in unobserved protein-space regions, extending natural-family repertoires through fine-tuning, and supplying learned representations to downstream engineering models.The authors present these as the first three of six direct applications.Source: Conclusion · p. 12
  • The proposed applications also include generating sequences with specific functional properties and designing novel, purpose-driven receptors and enzymes with encoder-decoder Transformers.These are the fourth and fifth applications in the authors’ ordering.Source: Conclusion · p. 12
  • Interpreting protein language models could clarify sequence-structure-function relationships and protein-folding rules; the authors expect these models to revolutionize protein design and offer solutions to societal challenges.They also hope the proposal encourages collaboration between Artificial Intelligence and Biochemistry communities.Source: Conclusion · p. 12

Related papers

Loading 2201.07338v2…