Source-linked AI summary
ProGen: Language Modeling for Protein Generation
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R. Eguchi, Po-Ssu Huang, Richard Socher
TL;DR
Protein engineering needs better ways to generate proteins with desired properties despite reliance on heuristic mutations and scarce structural annotations. ProGen addresses this with a tag-conditioned language model trained on 280 million sequences, generating proteins with near-native conformational energies and favorable structural metrics.
Problem
Protein engineering relies on heuristic mutations, while structural annotations remain scarce because obtaining three-dimensional structures is expensive and time consuming.
Method
ProGen is a 1.2B-parameter conditional language model trained on 280M protein sequences with taxonomic, functional, and locational conditioning tags.
Results
Generated proteins exhibit near-native conformational energies, while secondary-structure accuracy surpasses the 25% mutation baseline with at least 8 conditioning tags.
Takeaways & Limitations
ProGen may complement existing protein-engineering methods by providing conditioned candidate sequences, including for low-homology applications where existing techniques fall short.
Abstract
from arXiv · showhide
Generative modeling for protein engineering is key to solving fundamental problems in synthetic biology, medicine, and material science. We pose protein engineering as an unsupervised sequence generation problem in order to leverage the exponentially growing set of proteins that lack costly, structural annotations. We train a 1.2B-parameter language model, ProGen, on ~280M protein sequences conditioned on taxonomic and keyword tags such as molecular function and cellular component. This provides ProGen with an unprecedented range of evolutionary sequence diversity and allows it to generate with fine-grained control as demonstrated by metrics based on primary sequence similarity, secondary structure accuracy, and conformational energy.
1. Introduction
Protein engineering is constrained by heuristic sequence selection and a severe imbalance between abundant raw sequences and scarce structural annotations. ProGen addresses this gap with a large conditional language model that generates controllable protein sequences and demonstrates structural and functional promise.
- Motivation: Directed evolution still relies on heuristics and random mutations to select initial protein sequences for iterative optimization.Protein engineering has produced enzymes, therapies, and sensors, but sequence selection remains experimentally driven.
- Motivation: Three orders of magnitude more raw protein sequences exist than sequences with structural annotations, while sequence data grows near exponentially.Structural information is costly to obtain, motivating methods that learn from unannotated sequence data.
- Related work: Prior work adapted language-model representation learning to classify protein properties, but had not applied artificial text-generation methods to controllable protein generation.This establishes the methodological gap ProGen is designed to address.
- Contribution: 1.2 billion parameters and 280 million protein sequences define ProGen, a conditional language model trained with taxonomic, functional, and locational conditioning tags.The tags enable generation tailored to desired properties.
- Results: ProGen performs comparably to similarly sized English models, improves with larger amino acid contexts and more conditioning tags, and generates proteins with favorable structural metrics.Evaluation includes sequence similarity, secondary structure accuracy, and conformational energy; case studies include held-out VEGFR2 completion and high-fitness selection.
2. Related Work
Prior work applies contextual language-model representations to proteins and explores both structure-conditioned and sequence-only generative modeling. Large Transformer language models also motivate controllable generation through conditioning on readily extracted text properties.
- Protein representation learning: Contextualized language-model representations support protein representation learning, from which structural information can be extracted and representations adapted for other tasks.The passage frames methods from natural language processing as effective for contextual protein representations and cites Rives et al. (2019) as an example.
- Generative models for protein engineering: Structure-conditioned generative models use graph specifications, structural fragments, or explicit structural information, limiting their ability to capture sequence-only data diversity.Examples include transformer conditioning on graph-structured targets and GAN-based in-painting of missing residues in pairwise distance maps.
- Generative models for protein engineering: Sequence-only protein generation has also been explored with residual causal dilated convolutional neural networks and variational autoencoders.The passage contrasts these approaches with prior methods that explicitly use structural information.
- Language Models and Controllable Generation: Large Transformer models such as GPT-2 demonstrate strong unconditional language modeling and text generation after training on vast unsupervised English corpora.This establishes the language-modeling foundation for controllable generation discussed in the section.
3. Methods
ProGen frames controllable protein generation as conditional next-token prediction over amino-acid sequences and tags, implemented with a causal Transformer. It is trained on a large annotated protein corpus and evaluated using sequence, structural, and energy-based criteria with mutation and alanine baselines.
- Conditional formulation: Protein generation is formulated as next-token prediction over sequences formed by prepending conditioning tags to amino-acid sequences.The model learns p(x) over combined tag–protein sequences, recovering conditional distributions such as p(a|c).
- Conditional formulation: Desired proteins are generated by sequentially sampling amino acids conditioned on a specified tag sequence and previously generated residues.At each step, the model samples from the distribution for the next token and updates the context until the desired length is reached.
- Architecture and optimization: The model uses a causal Transformer with token and sinusoidal positional embeddings, multi-head attention, feedforward ReLU blocks, normalization, residual connections, and softmax sampling.Training uses cross-entropy loss over output scores, while generation normalizes final-token scores with a softmax.
- Data and training: The aggregated training corpus contains over 281M proteins from multiple annotated sequence and taxonomy databases.Training includes each sequence and its reverse, prepends sampled conditioning-tag subsets, and biases tag selection toward manually verified SWISSPROT annotations.
- Evaluation: Generative quality is assessed through primary sequence similarity, secondary-structure accuracy, and conformational-energy analysis against randomized-mutation and all-alanine baselines.The evaluation also measures perplexity, hard accuracy, and BLOSUM62-based soft accuracy for modeling training and test distributions.
4. Results and Analysis
ProGen generalizes across protein sequences and unseen families, with performance improving as amino-acid and conditioning-tag context increases. As a generative model, it produces sequences with strong structural and energetic properties, including function-preserving substitutions and low-energy completions.
- Language-model evaluation: ProGen achieves perplexities representative of a high-quality language model and dramatically improves over Uniform and Empirical Baselines on the full test set.The supplied passage does not report the specific Table 1 perplexity values.
- Language-model evaluation: ProGen generalizes to unseen protein families, outperforming the Empirical Baseline on OOD-test sequences despite worse performance than on ID-test sequences.Fine-tuning on OOD-test-80 improves performance on OOD-test-20 over the same architecture trained from random initialization.
- Language-model evaluation: Perplexity decreases and hard accuracy increases for later protein segments, while the same trends hold when increasing conditioning-tag context.Figure 3 examined intervals up to 500 tokens with at least 30k samples per interval; Figure 4 examined up to 14 conditioning tags with at least 3k samples per category.
- Generation quality: More than 20% higher BLOSUM62 soft accuracy than hard accuracy indicates that ProGen errors often follow natural substitutions preserving higher-level structure.The result suggests ProGen learned function-preserving mutational invariances despite modeling proteins as sequences.
- Generation quality: With k = 1 and a repetition penalty, ProGen generates proteins with sequence similarity comparable to randomly mutating 50% of amino acids outside the given context.These nearly greedy settings were used for subsequent generation experiments.
- Generation quality: 73.1% mean sequence identity accompanies better Rosetta energies than the 25% mutation baseline, while ProGen completions approach or fall below relaxed native-template energy levels.In VEGFR2 completion experiments, ProGen samples had lower conformational energy than all baselines, indicating better structural conservation.
5. Conclusion
ProGen is a controllable protein-generation language model trained on broad evolutionary sequence diversity. Its near-native structure energies suggest functional viability and a complementary role in protein engineering, including domains where existing techniques fall short.
- ProGen is a controllable protein-generation language model trained on the full evolutionary diversity of one of the largest sequence databases.
- Near-native structure energies likely imply that ProGen’s generated proteins are functionally viable.
- ProGen could complement state-of-the-art protein-engineering methods, including directed-evolution data augmentation for non-homologous domains where MSAs fall short.
A. Appendix · A.1. Measuring out-of-distribution
The appendix evaluates ProGen with in-distribution and out-of-distribution test subsets. OOD-Test is constructed from held-out Pfam families and compared with Train and ID-Test using normalized 3-gram and 5-gram histogram divergence.
- A.1. Measuring out-of-distribution: The evaluation uses two test subsets: ID-Test, a random split of the non-redundant sample database, and OOD-Test, an out-of-distribution set.ID-Test contains held-out samples from the typical in-distribution setting.
- A.1. Measuring out-of-distribution: OOD-Test contains samples with matching subsequences in one of twenty Pfam protein families excluded from Train and ID-Test.The held-out families define the out-of-distribution evaluation set.
- A.1. Measuring out-of-distribution: SAE is the sum of absolute errors between paired normalized 3-gram and 5-gram histograms.SAE measures distributional difference between dataset pairs.
- A.1. Measuring out-of-distribution: Train and ID-Test appear similarly distributed, while OOD-Test is markedly different from both.The histogram-based results support OOD-Test as a substantially distinct evaluation distribution.
- A.1. Measuring out-of-distribution: The analysis compares normalized 3-gram and 5-gram histograms across Train, ID-Test, and OOD-Test.The histograms are computed across samples in each dataset.
- A.1. Measuring out-of-distribution: 0 is the SAE for perfectly aligned normalized histograms, whereas 2 is the SAE for completely divergent histograms.These values define the interpretation range of the sum of absolute errors.
A.2. Generation with only conditioning tags
ProGen can generate protein sequences from conditioning tags alone, without an initial amino acid context. A 400-residue sequence generated with Flavoprotein and FMN tags produced multiple strong HHblits alignments to related sequences.
- Tag-only generation: ProGen greedily generates protein sequences using only conditioning tags, demonstrated with the Flavoprotein and FMN tags and no initial amino acid context.The FMN tag denotes proteins involved in FMN synthesis or containing FMN as a prosthetic group, cofactor, or cosubstrate.
- Alignment analysis: 400 residues were generated and searched with HHblits, yielding multiple sequences that aligned well with the ProGen sequence.Figures 14–16 indicate high E-values and related properties; lower E-values correspond to a lower probability of random matching.
A.3. Model visualizations
ProGen’s learned representations align with known amino-acid properties and reveal differentiated attention patterns. Generated FMN sequences also align with multiple sequences having properties reflective of FMN proteins, including oxidoreductases.
- A.3. Model visualizations: PCA of ProGen’s trained amino-acid embeddings aligns with intuitive biophysicochemical properties despite random initialization without prior residue-property knowledge.The embeddings were learned through per-token training on millions of protein sequences.
- A.3. Model visualizations: ProGen’s attention heads exhibit well-differentiated local and global patterns that may indicate specialization on different tasks.The visualization covers layers 1–3 and attention heads 1–12 for α-actinin residues 510–528, which exhibit an alpha-helical structure.
- A.3. Model visualizations: Multiple sequences align well with a ProGen-generated FMN sequence conditioned only on tags, with matching alignments reflecting FMN protein properties such as oxidoreductase function.The multiple sequence alignment was obtained directly using HHblits; significantly low E-values indicate matching homologs.
- A.3. Model visualizations: Three ranked alignments of a ProGen-generated FMN protein use E-value < 1e−4 and identity > 40% as thresholds for potentially homologous matches.In these alignments, Q labels the ProGen protein and T labels the matched sequence.
- A.3. Model visualizations: Two example attention heads show local attention to previous tokens for each predicted token.These corresponding heads are shown as local attention patterns in Figure 18.