Source-linked AI summary
Protein sequence design with deep generative models
Zachary Wu, Kadina E. Johnston, Frances H. Arnold, Kevin K. Yang
TL;DR
Protein engineering needs efficient ways to identify sequences with improved properties, while available protein sequences are often not paired with consistent fitness measurements. This review surveys deep generative-model applications for learning representations, generating sequences, and optimizing them with experimental feedback. The field has shown preliminary success across diverse settings, but its optimization frameworks and data foundations remain limited.
Problem
Protein engineering seeks improved sequences, but large sequence databases are rarely coupled to fitness measurements collected under consistent experimental conditions.
Method
The review synthesizes deep generative-model applications for downstream prediction, protein-sequence generation, and optimization, including conditioning and reinforcement-learning approaches.
Results
Deep generative models have shown preliminary success generating useful protein sequences across varied scales and engineering settings.
Takeaways & Limitations
Integrating machine learning with experimental rounds can help protein engineers efficiently search sequence space and maximize the value of expensive laboratory work.
Takeaways & Limitations
The review excludes structure-informed design, while generative optimization remains immature and lacks a generally advantageous framework.
Abstract
from arXiv · showhide
Protein engineering seeks to identify protein sequences with optimized properties. When guided by machine learning, protein sequence generation methods can draw on prior knowledge and experimental efforts to improve this process. In this review, we highlight recent applications of machine learning to generate protein sequences, focusing on the emerging field of deep generative methods.
1. Introduction
This review frames deep generative models as tools for protein engineering that learn from unlabeled sequences and support representation learning, sequence generation, and optimization. It focuses on three applications while excluding structure-informed design.
- Machine learning-guided protein engineering combines experimental information with data-driven models to find desirable protein sequences more efficiently.
- Structure-informed protein design is outside the review’s scope, which is limited to deep generative models of protein sequence.
- Deep generative models learn underlying protein-sequence distributions rather than mappings from inputs to labels.
- Their three fundamental tasks are representation learning, generating unseen sequences, and assigning higher likelihood to sequences meeting desired criteria.
- The review covers pretrained representations for downstream prediction, sequence generation, and model optimization toward desirable properties.
2. Fine-tuning on downstream tasks
Downstream protein-engineering workflows use generative models to learn sequence representations or fine-tune pretrained models before discriminative prediction. These representations can compress sequence information, but their advantage over simpler encodings depends on available training data.
- Discriminative protein-engineering workflows train regression models on measured sequence–function pairs to estimate experimental values and search for improved sequences.
- Generative models encode discrete protein sequences in continuous, compressed latent spaces that may capture contextual information for downstream modeling.
- BioSeqVAE learned representations from 200,000 SwissProt sequences and supported random-forest classification of localization and enzyme class.
- Autoregressive models pretrained on large protein databases outperform other architectures on several downstream discriminative tasks, and double fine-tuning can find improved sequences from few measured sequences.
3. Protein sequence generation
Deep generative models have produced experimentally tested protein sequences across VAEs, GANs, autoregressive models, and Transformers. Conditioning generation on task information can increase desired sequence features and recover specific binding motifs.
- VAEs, GANs, autoregressive models, and related architectures have been used to generate novel functional protein sequences.
- VAEs generated functional luciferases using either aligned-sequence training or an autoregressive decoder for unaligned sequences.
- 24% of 20,000 ProteinGAN sequences displayed near-wild-type malate dehydrogenase activity, including a variant with 106 mutations from the closest known sequence.
- Autoregressive models generated single-domain antibodies without requiring sequence alignment, which is advantageous for highly variable antibody regions.
- A Transformer trained on 25,000 target–signal-peptide pairs produced signal peptides, with roughly half of tested sequences yielding secreted and functional enzymes in Bacillus.
- Conditioning a VAE decoder on metal cofactors increased desired metal-binding sites, and 11% of 1,000 sequences recovered the correct amino-acid triad for a removed copper-binding site.
4. Optimization with Generative Models
Generative-model optimization shifts from producing merely viable sequences to assigning greater probability to sequences with improved properties. Approaches include biased GAN retraining, adaptive sampling, and reinforcement learning, but no framework yet has clear general advantages.
- Generative-model optimization trains models to place higher probability on improved protein sequences rather than merely generating viable examples.
- Biasing Wasserstein GAN training data toward desired antibody properties shifted the estimated properties of generated sequences in the desired direction.
- Design by Adaptive Sampling weights sequences by their probability of exceeding the previous iteration’s Qth-percentile score, enabling stricter optimization and convergence under some conditions.
- In reinforcement-learning sequence design, amino acids are selected left to right as actions, with completed-sequence fitness serving as the final reward.
- A panel of surrogate fitness models that pass cross-validation accuracy thresholds is used to estimate rewards for updating the policy network.
- This optimization direction remains in its infancy, with no clear generally advantageous framework and an unresolved balance between sequence diversity and trusted sequence-space regions.
5. Conclusions and Future Directions
Machine learning has shown preliminary success in protein engineering, generating useful or optimized sequences across scales while expanding beyond nature’s observed mutations. Progress remains constrained by limited, heterogeneous fitness data and the need for careful integration of experimental measurements.
- Machine learning enables access to optimized protein sequences with unprecedented efficiency and sampling beyond nature’s repertoire of mutations.
- Many demonstrated approaches still require testing many protein variants, making data collection a major driver of progress.
- Protein sequence databases are extensive but rarely pair sequences with fitness measurements collected under consistent experimental conditions.
- Deep mutational scanning and continuous evolution can collect larger datasets, but their artifacts and cross-study differences require careful consideration.
- Data-driven protein engineering combines machine learning with experimentation to design useful sequences across low-, medium-, and higher-throughput settings.
Appendix A. Appendix: Deep Generative Models of Protein Sequence
The review introduces three popular deep generative model classes for protein sequences: variational autoencoders, generative adversarial networks, and autoregressive models.
- The review describes variational autoencoders, generative adversarial networks, and autoregressive models as applications to protein sequences.
Appendix A.1. Variational Autoencoders
Variational autoencoders encode protein sequences into lower-dimensional latent representations and decode those representations to reconstruct or generate sequences. Their objective balances reconstruction accuracy with regularization, producing a structured latent space useful for downstream prediction and interpolation.
- An encoder q(z|x) maps each protein sequence into a comparatively low-dimensional latent representation, while a decoder reconstructs the input.
- The prior over z is usually normal(0, 1), and direct evaluation of the marginal likelihood integral is replaced by a variational lower bound.
- The VAE objective combines reconstruction accuracy with a Kullback-Leibler divergence regularization term measuring information lost when q represents p.
- The encoder supports downstream prediction, while the decoder generates new examples as nonlinear interpolations of training examples.
- The prior over z enables smooth interpolation between latent points and imposes structure on the representation.
Appendix A.2. Generative Adversarial Networks
Generative adversarial networks generate synthetic protein-sequence examples through an adversarial interaction between a generator and discriminator. Their training objective is a minimax equilibrium, but reaching that equilibrium can be difficult in practice.
- A GAN generator G maps random noise to data-space examples, while discriminator D distinguishes generated examples from real data.
- The generator and discriminator are trained through a minimax game in which the discriminator separates real and generated examples.
- GANs do not perform representation learning or density estimation, although they often generate more realistic image examples than VAEs.
- The Nash equilibrium between generator and discriminator networks can be notoriously difficult to obtain in practice.
Appendix A.3. Autoregressive models
Autoregressive models generate sequences token by token from preceding tokens, while masked language models reconstruct hidden elements from surrounding context. These models learn sequence distributions and contextual representations, but capturing long-range dependencies remains challenging.
- Autoregressive models generate each subsequent token conditioned on previously generated tokens.
- Sequence probability is factorized into a product of conditional distributions over successive positions.
- Masked language models replace sequence elements with mask tokens and learn to reconstruct them from the remaining sequence.
- Autoregressive models maximize training-sequence probability, generate new sequences, and can provide contextual representations for sequence positions.
- Capturing long-range dependencies is the main challenge, addressed differently by dilated convolutional, recurrent, and Transformer-based architectures.