Source-linked AI summary
ProGen2: Exploring the Boundaries of Protein Language Models
Erik Nijkamp, Jeffrey Ruffolo, Eli N. Weinstein, Nikhil Naik, Ali Madani
TL;DR
Protein language models have succeeded in protein tasks, but the roles of very large models and data distributions remain insufficiently understood. ProGen2 addresses this gap with models up to 6.4B parameters trained on datasets spanning over a billion proteins. The models capture natural sequence distributions, generate viable sequences, and predict fitness without additional fine-tuning, while the results emphasize data distribution and alignment for functional prediction.
Problem
The study addresses limited understanding of how very large protein language models and training-data distributions affect protein engineering and design.
Method
The authors train ProGen2 autoregressive protein language models up to 6.4B parameters on sequence datasets from genomic, metagenomic, and immune repertoire databases.
Results
ProGen2 captures observed evolutionary sequence distributions, generates viable sequences, and predicts protein fitness without additional fine-tuning.
Takeaways & Limitations
The results indicate that protein sequence models should place greater emphasis on the data distribution used for training and its alignment with desired functional prediction.
Takeaways & Limitations
Broad adoption of sequence fitness prediction and natural-protein distribution modeling requires oversight of end uses and downstream effects to support safe, ethical applications.
Abstract
from arXiv · showhide
Attention-based models trained on protein sequences have demonstrated incredible success at classification and generation tasks relevant for artificial intelligence-driven protein design. However, we lack a sufficient understanding of how very large-scale models and data play a role in effective protein model development. We introduce a suite of protein language models, named ProGen2, that are scaled up to 6.4B parameters and trained on different sequence datasets drawn from over a billion proteins from genomic, metagenomic, and immune repertoire databases. ProGen2 models show state-of-the-art performance in capturing the distribution of observed evolutionary sequences, generating novel viable sequences, and predicting protein fitness without additional finetuning. As large model sizes and raw numbers of protein sequences continue to become more widely accessible, our results suggest that a growing emphasis needs to be placed on the data distribution provided to a protein sequence model. We release the ProGen2 models and code at https://github.com/salesforce/progen.
1 Introduction
The paper frames protein engineering as important but constrained by reliance on directed evolution, and investigates whether larger protein language models and broader data improve protein design. ProGen2 scales models and datasets while evaluating generation and zero-shot fitness prediction.
- Protein engineering supports applications in pharmaceuticals, agriculture, specialty chemicals, and fuel, but current tools mainly rely on iterative directed evolution.Directed evolution stochastically mutates a starting sequence, measures variants, and iterates toward improved fitness.
- The study examines how very large-scale models and data affect protein language model development.
- ProGen2 models range from 151M to 6.4B parameters and use datasets collectively totaling 1B protein sequences from genomic, metagenomic, and immune repertoire databases.
- The authors analyze generated sequences through predicted structural and biophysical properties.
- The study evaluates fitness prediction on existing experimental datasets to investigate the roles of data distribution and alignment in protein language modeling.
2 Related Work
Prior work uses language modeling and alignment-based methods for protein generation and fitness prediction, but each approach has distinct scope and data challenges. ProGen2 adopts causal autoregressive modeling to learn from broad protein sequence collections.
- Causal language models represent sequence likelihood as a product of next-token conditionals, enabling tractable sampling and likelihood estimation.
- Large language models motivate protein applications through scaling laws and few-shot task capabilities that can emerge without task-specific parameter fine-tuning.
- ProGen2 formulates conditional protein generation as autoregressive next-token prediction over raw amino acid sequences.
- Protein language models learn across families to generate diverse, realistic proteins that can be structurally well-folded despite sequence divergence.
- Deep mutational scanning measures mutation effects experimentally, but typically requires thousands of variants and remains limited relative to the full combinatorial mutation space.
- Alignment-based fitness models exploit covariation but are limited by shallow MSAs, difficult antibody alignment, and indels that complicate evaluation.
3 Methods
ProGen2 uses autoregressive transformer decoders trained at multiple scales on universal, family-specific, and antibody sequence datasets. The study samples proteins across generation settings and evaluates language modeling, structural properties, and zero-shot fitness.
- 3.1 Model architecture: ProGen2 models are autoregressive transformers trained with next-token prediction at 151M, 764M, 2.7B, and 6.4B parameters.
- 3.1 Model architecture: The architecture is a causal transformer decoder with left-to-right masking, rotary positional encodings, and parallel self-attention and feed-forward computation.
- 3.2 Data: Standard models use mixtures of Uniref90 and BFD30, while PROGEN2-BFD90 mixes Uniref90 with representative sequences from several clustered databases.
- 3.2 Data: PROGEN2-OAS is trained on 554M redundancy-reduced antibody sequences from the 1.5B-sequence Observed Antibody Space database.
- 3.4 Generation experiments: The study samples universal, fold-specific, and antibody sequences, including 5,000 universal sequences and 52K unprompted antibody sequences.
- 3.5 Evaluation: Evaluation uses held-out sequence test sets at 90% and 50% maximum identity thresholds, plus narrow, wide, and antibody-specific experimentally measured fitness landscapes.
4 Results
PROGEN2 models capture natural protein sequence distributions increasingly well with scale, generate structurally plausible novel proteins, and show that training-data distribution can matter more than model size for zero-shot fitness prediction. Generation behavior depends on finetuning and sampling choices, while antibody generation benefits from filtering and less restrictive sampling.
- 4.1 Capturing the distribution of observed proteins: Larger PROGEN2 models achieve substantially lower perplexity on held-out natural sequences, indicating improved modeling of the training distribution.This measures sequence-distribution capture rather than molecular fitness.
- 4.2 Generation ability: Two epochs of finetuning on 1M two-layer sandwich proteins produced sequences more similar to natural proteins, while sampling temperature and nucleus probability controlled sequence and structural diversity.Lower sampling temperature increased structural similarity, whereas higher temperature or nucleus probability increased sequence diversity.
- 4.2 Generation ability: 5,000 PROGEN2-xlarge generations had median TMscore 0.89 and median pLDDT 73.7, with many structural homologs in the PDB despite low sequence identity in some cases.One solenoid example shared only 11.2% sequence identity with its closest structural homolog, while another multi-domain sequence had no PDB structural match.
- 4.2 Generation ability: Less restrictive sampling generated antibody sequences with lower aggregation propensity and higher solubility, and PROGEN2-base ranking further improved solubility but only marginally improved aggregation propensity.These results support generating broadly and filtering with a universal protein language model for antibody libraries.
- 4.3 Zero-shot fitness prediction: PROGEN2-small outperformed the order-of-magnitude-larger RITA-XL on average narrow-landscape zero-shot fitness prediction, while PROGEN2-base peaked at 764M parameters before larger models declined.The models matched or outperformed baselines across modeling strategies, highlighting the role of training-sequence distribution.
- 4.3 Zero-shot fitness prediction: Larger models may help on wider, low-homology landscapes such as epistatic GB1, whereas antibody-specific OAS training performed poorly relative to universal-protein pretraining for general antibody fitness properties.The largest models may show emergent behavior for identifying top-ranked GB1 variants, while universal models better predict general antibody properties than binding affinity.
5 Conclusion
The results support continued scaling of protein language models while emphasizing alignment between training data distributions and desired functional prediction. The authors also highlight ethical oversight for protein-design applications.
- Models may continue scaling beyond 6B parameters, with improvements in fitting natural-sequence distributions.
- Large models can generate viable sequences that expand natural protein sequence and structural space, especially for difficult or out-of-distribution problems.
- Zero-shot fitness results indicate that training-data alignment with desired functional prediction matters, while average Spearman fitness has evaluation biases.
- Protein-design systems require active oversight during project initiation, experimental optimization, and deployment to limit harmful effects.
A Supplementary Material
The supplementary material reports additional zero-shot fitness-prediction results across indel, antibody, and narrow-landscape studies, plus comparisons involving metagenomic data distributions.
- Table 6 reports zero-shot fitness prediction on experimentally measured indel landscapes, with PROGEN2 outperforming baselines including retrieval-based models.
- Table 7 provides full zero-shot fitness-prediction results for the antibody landscapes evaluated in the study.
- Table 8 compares language modeling and zero-shot fitness prediction for two 2.7B-parameter models trained with different amounts of metagenomic protein data.BFD30 and BFD90 are clustered at 30% and 90% sequence identity; Uniref90+BFD90 is majority metagenomic.
- Table 9 presents full zero-shot fitness-prediction results for narrow fitness landscapes.