Source-linked AI summary
Controllable Protein Design with Language Models
Noelia Ferruz, Birte Höcker
TL;DR
Protein design faces major environmental and medical needs, while current approaches remain limited and protein sequence–structure–function relationships are difficult to exploit directly. The paper surveys how NLP and Transformer methods can be transferred to protein sequences, including pretrained, conditional, and interpretable models. Early applications include improved protein predictions and experimentally validated generated enzymes, but biological and computational constraints remain.
Problem
Current methods cannot yet design proteins as proficient as natural ones, and applying NLP is challenged by unclear protein token boundaries and difficult functional interpretation.
Method
The paper surveys NLP-to-protein transfer, emphasizing Transformer architectures, pretrained language models, conditional control tags, and interpretability methods for protein design.
Results
Early applications produced improved mutational-effect and long-range-contact predictions, while ProGen-generated lysozymes showed natural-range enzymatic activity and one variant recapitulated the native three-dimensional structure.
Takeaways & Limitations
Protein language models are a promising and largely unexplored approach for generating functional and controllable protein sequences and studying sequence–structure–function relationships.
Takeaways & Limitations
Protein NLP remains constrained by difficult sequence interpretation and by the economic and computational accessibility of large models.
Abstract
from arXiv · showhide
The 21st century is presenting humankind with unprecedented environmental and medical challenges. The ability to design novel proteins tailored for specific purposes could transform our ability to respond timely to these issues. Recent advances in the field of artificial intelligence are now setting the stage to make this goal achievable. Protein sequences are inherently similar to natural languages: Amino acids arrange in a multitude of combinations to form structures that carry function, the same way as letters form words and sentences that carry meaning. Therefore, it is not surprising that throughout the history of Natural Language Processing (NLP), many of its techniques have been applied to protein research problems. In the last few years, we have witnessed revolutionary breakthroughs in the field of NLP. The implementation of Transformer pre-trained models has enabled text generation with human-like capabilities, including texts with specific properties such as style or subject. Motivated by its considerable success in NLP tasks, we expect dedicated Transformers to dominate custom protein sequence generation in the near future. Finetuning pre-trained models on protein families will enable the extension of their repertoires with novel sequences that could be highly divergent but still potentially functional. The combination of control tags such as cellular compartment or function will further enable the controllable design of novel protein functions. Moreover, recent model interpretability methods will allow us to open the 'black box' and thus enhance our understanding of folding principles. While early initiatives show the enormous potential of generative language models to design functional sequences, the field is still in its infancy. We believe that protein language models are a promising and largely unexplored field and discuss their foreseeable impact on protein design.
Introduction
Protein design aims to create custom structures with desired functions, but current methods remain far from matching naturally generated proteins. The inverse folding problem requires finding sequences that adopt target folds, while protein sequences themselves offer a route for applying NLP methods.
- Introduction: Despite major advances, contemporary research remains far from designing proteins as proficient as naturally generated ones.
- Introduction: Protein design seeks sequences that adopt structures with desired functions, formalizing the task as an inverse protein folding problem.Optimization algorithms search a sequence–structure landscape for an energy minimum.
- Introduction: Protein sequences encode structural and functional properties efficiently, even without explicitly imposing biophysical constraints.This makes sequence data attractive for importing Natural Language Processing methods into protein research.
The language of proteins
Protein sequences share hierarchical and relational features with human language, motivating NLP-based approaches, but important biological differences complicate the analogy. These differences include unclear token boundaries, limited interpretability of generated sequences, and distinct evolutionary processes.
- The language of proteins: Amino acids form structural elements, fragments form larger structures, and protein assemblies parallel the hierarchical organization of letters, words, sentences, and texts.
- The language of proteins: Protein sequences exhibit local and long-distance dependencies because distant amino acids can interact within three-dimensional structures.
- The language of proteins: Protein token boundaries are less discernible than word boundaries and may require structural information or computationally intensive calculations.
- The language of proteins: Generated protein sequences remain difficult to interpret functionally and require extensive experimental tests, despite available sequence corpora.
- The language of proteins: These similarities motivate NLP perspectives on protein design even though the differences pose considerable challenges to applying NLP methods.
NLP has had an impact on protein research for decades
NLP techniques have influenced protein research for decades, progressing from shallow machine-learning methods to neural networks and newer pretrained language models. This history supports adapting pretrained models to protein sequences and design.
- NLP has had an impact on protein research for decades: NLP methods such as Support Vector Machines and Hidden Markov Models were transferred to protein classification, labeling, fold recognition, and related tasks.
- NLP has had an impact on protein research for decades: The glossary defines autoencoding models as predicting original inputs after masking or corrupting some tokens.
- NLP has had an impact on protein research for decades: Convolutional networks extended from NLP tasks to protein disorder prediction, DNA-binding-site prediction, and fold classification.
- NLP has had an impact on protein research for decades: The growth of modern NLP has created substantial potential for adapting pretrained language models to protein research and design.
- NLP has had an impact on protein research for decades: NLP development has repeatedly preceded corresponding applications in protein research, as summarized by the field's parallel timelines.
Attention mechanism and Transformers
Transformers use attention to process sequence-wide information more effectively and enable parallel training, with encoder-only, decoder-only, and encoder–decoder variants. Pretraining followed by fine-tuning made GPT especially suited to language generation, while BERT primarily produces representations for downstream tasks.
- Attention mechanism and Transformers: Attention lets a decoder analyze the whole input and focus on relevant parts, addressing sequence-length-related degradation and training costs in recurrent seq2seq models.
- Attention mechanism and Transformers: The original Transformer uses stacked encoder and decoder layers with self-attention, feed-forward processing, and encoder–decoder attention that can run in parallel.
- Attention mechanism and Transformers: Dot-product attention converts inputs into embeddings with positional information, derives Query, Key, and Value vectors, and combines scored Values into output vectors.
- Attention mechanism and Transformers: Table 1 summarizes Transformer sizes and configurations using N for layers, H for attention heads, and d for model dimension.
- Attention mechanism and Transformers: GPT pretrains on unlabeled text, then fine-tunes on labeled tasks; its autoregressive next-item objective makes it particularly powerful for generation.
- Attention mechanism and Transformers: GPT uses the Transformer decoder, whereas BERT uses the encoder and is primarily applied to create representations for downstream tasks.
Protein sequences are ideal candidates for Transformers
Transformer models benefit from expanding digital corpora, and protein sequence databases provide especially large and rapidly growing datasets relative to structural databases.
- Transformer success is linked to exponentially growing training corpora that support larger and more powerful models.
- Protein sequence characterization is growing faster than structural characterization, creating an expanding sequence-structure gap.UniParc contained 441,169,278 entries in release 2021_03, while functional annotation remains time-consuming.
- Figure 4 compares prominent Transformers by parameter count with deposited entries in the PDB and UniParc protein databases.
Transformers for protein design
Protein language models progressed from early sequence models to pretrained Transformers whose representations support downstream prediction, while publicly available protein sequence generation remained unavailable at the time described.
- Early protein language models used n-grams and LSTMs to generate probabilistic models or vector representations of protein sequences.
- ESM-1b was the first Transformer pretrained on protein data, using 250 million sequences to produce protein representations.Its architecture and training objective followed BERT.
- ESM-1b representations encode protein organization from amino-acid biochemical properties to evolutionary relationships among proteins.Coupled downstream models enabled mutational-effect and long-range-contact predictions.
- ProtTrans adapted six Transformer architectures to proteins using more than 390 billion amino acids from large sequence databases.
- Before the reported generation initiatives, no publicly available pretrained models for protein sequence generation existed.
- GPT-style finetuning uses pretrained representations coupled to task-specific models for predicting protein properties such as stability changes or function loss.
Tailored protein design
Conditional Transformers extend protein language models from sequence generation toward controllable design by using property tags and family-specific finetuning, with early experimental evidence from generated lysozymes.
- Fine-tuning ProGen on five protein families produced lysozyme sequences with enzymatic activities in the range of natural lysozymes.
- X-ray characterization of one generated lysozyme variant showed that it recapitulated the native three-dimensional structure.
- Conditional Transformers are presented as a research direction for controlled generation of protein sequences and potentially their properties.
- Proposed controls include generating proteins that bind ATP, form all-beta structures, or are membrane-bound.
- Figure 5 maps Transformer-enabled generation, finetuning, downstream prediction, property control, protein-space exploration, interpretability, and receptor or enzyme design.
Enzyme, receptor, and biosensor design
Encoder-decoder Transformers can condition ligand generation on protein sequences, while reversing or extending this translation framework is proposed for receptor, biosensor, and enzyme design.
- The Molecular Transformer predicts chemical reaction products from reactants and reagents, and its reversal frames retrosynthesis as finding reactants for a target molecule.
- An encoder-decoder model can process protein sequences and generate SMILES for ligands potentially compatible with binding the input sequence.
- Reversing this translation setup is proposed to generate protein sequences compatible with input SMILES for receptor and biosensor design.
- A model mapping chemical reactions to protein sequences is proposed as a route to enzyme design, including enzymes for reactions not found in nature.The passage connects this possibility to biological strategies for reversing environmental pollution.
Explainable protein design
Explainable AI offers ways to interpret protein-design models and investigate sequence–structure–function relationships, while access to increasingly large Transformer models remains concentrated among major companies.
- Explainable AI aims to clarify why deep learning models reach particular answers and can lead scientists toward new ideas and approaches.
- Understanding model representations could help reveal sequence–structure–function relationships and principles governing protein folding.
- Transformer attention provides an importance score over input features that can be visualized as a saliency heatmap.
- Training increasingly capable Transformer models may exceed the economic and environmental resources available to startups and academic research groups.The cited discussion identifies both training cost and carbon footprint as concerns, while noting that larger models generally perform better.
- Protein Transformer development has so far involved large companies, but public releases and academic collaborations could still extend benefits to the research community.
Conclusion
Transformer-based language models open new directions for protein research and customizable protein design. The paper outlines applications ranging from sequence generation and family extension to conditional design and model interpretation.
- Recent NLP advances, especially Transformer models, create opportunities to apply language-model capabilities to protein sequence research and design.
- The paper identifies six applications, including generating unexplored sequences, extending protein-family repertoires, and using learned representations for downstream engineering.
- The proposed applications also include conditional functional sequences, purpose-driven receptors and enzymes, and interpretable models of folding principles.