PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Generative Modeling for Protein Design

Alexey Strokach, Philip M. Kim

arXiv:2109.13754v1cs.LGcs.AI

TL;DR

Protein design faces challenges including optimizing existing proteins and generating new proteins with specific shapes and functions. This review describes five classes of generative models and a framework for designing proteins to meet objectives. Across reviewed scenarios, these models produced novel proteins, often orders of magnitude faster than traditional approaches.

  • Problem

    Protein design still faces challenges in optimizing existing proteins and generating new ones with specific shapes and functions.

    Source: §1 · p. 3

  • Method

    The review describes five classes of generative models and strategies combining them with supervised models, simulations, and domain expertise to produce proteins meeting specific objectives.

    Source: §1 · p. 5

  • Results

    Across reviewed scenarios, deep generative models produced novel proteins, often orders of magnitude faster than traditional approaches.

    Source: §4 · p. 11

  • Takeaways & Limitations

    The continued growth of protein sequence and structure data, alongside protein-specific libraries and network architectures, is likely to bring further improvements.

    Source: §4 · p. 11

    Generated examples can be of low diversity (mode collapse).

    Source: §2.1 · p. 7

Abstract

from arXiv · show

Deep learning approaches have produced substantial breakthroughs in fields such as image classification and natural language processing and are making rapid inroads in the area of protein design. Many generative models of proteins have been developed that encompass all known protein sequences, model specific protein families, or extrapolate the dynamics of individual proteins. Those generative models can learn protein representations that are often more informative of protein structure and function than hand-engineered features. Furthermore, they can be used to quickly propose millions of novel proteins that resemble the native counterparts in terms of expression level, stability, or other attributes. The protein design process can further be guided by discriminative oracles to select candidates with the highest probability of having the desired properties. In this review, we discuss five classes of generative models that have been most successful at modeling proteins and provide a framework for model guided protein design.

Highlights

The highlights frame machine learning as a key component of protein design: generative models propose novel sequences and structures, conditioning targets specific properties, and discriminative oracles further fine-tune design.

  • Machine learning is becoming a key component of the protein design process.Source: Highlights · p. 2
  • Deep generative models can produce novel protein sequences and structures.Source: Highlights · p. 2
  • Conditioned generative models can produce proteins with specific properties.Source: Highlights · p. 2
  • Discriminative oracles can further fine-tune the protein design process.Source: Highlights · p. 2

1. Introduction

Protein design still involves challenges such as optimizing existing proteins and generating new ones with specific shapes and functions. This review organizes machine-learning approaches into a framework for designing proteins to meet specific objectives.

  • Protein design challenges include optimizing existing proteins and generating new proteins with specific shapes and functions.Traditional computational design uses sampling techniques to explore energy landscapes defined by molecular mechanics force fields or semi-empirical energy functions.Source: §1 · p. 3
  • With protein sequence and structural information continuing to grow, machine learning is poised to become indispensable for efficient and successful protein design.Source: §1 · pp. 3, 5
  • Figure 1 distinguishes generative models that propose proteins, discriminative models that predict properties, and design guided by discriminative oracles.Source: Figure 1 · p. 4
  • The review formalizes machine-learning algorithms for protein design and delineates a framework for generating proteins that meet specific objectives.It describes five classes of generative models and strategies combining generative models with supervised models, simulations, and domain expertise.Source: §1 · p. 5

2. Generative models of protein sequences and structures

The review surveys five generative model classes for protein sequences and structures, emphasizing their different trade-offs and applications to protein design.

2.1. Autoregressive models

  • Transformers generally outperform RNNs on reconstruction and downstream tasks; structural and function conditioning extend autoregressive models to structure-aware and targeted sequence design.Structured Transformer assigns native amino acids higher probabilities than sequence-only models; conditional transformers generated experimentally validated lysozymes.Source: §2.1 · pp. 5, 8

2.2. Variational autoencoders (VAEs)

  • VAEs support topology- or function-conditioned sequence design and generation of protein backbones with desired shapes.Reported examples include often-luminescent generated luciferases, antimicrobial peptides, and immunoglobulin backbones with expected geometric features.Source: §2.2 · pp. 8–9

2.3. Normalizing flows (NFs)

  • Normalizing flows enable exact training-data probabilities, and Boltzmann generators model protein dynamics by generating conformations and transitions between known states.Generated conformations were confirmed by molecular dynamics simulations, and the models captured energy differences between known states.Source: §2.3 · p. 9

2.4. Energy-based models (EBMs)

  • EBMs learn representations for protein sequences and structures and support fixed-backbone rotamer design, though evaluating every possible rotamer makes inference relatively slow.MaSIF fingerprints enabled faster protein-interaction prediction and docking at comparable accuracy; Atom Transformer achieved accuracy comparable to Rosetta in recovering native rotamers.Source: §2.4 · pp. 9–10

2.5. Generative adversarial networks (GANs)

  • GANs generate or refine protein distance matrices and generate sequences with specific folds or functions.In one experimental example, generated malate dehydrogenase sequences possessed enzymatic activity in approximately 24% of cases.Source: §2.5 · p. 10

3. Model guided protein design

Model-guided protein design combines a pretrained generative model to propose candidates with discriminative oracles to validate, prioritize, or steer them toward desired properties.

  • A pretrained generative model, preferably conditioned on target structure or function, generates the initial candidate pool.Source: §3 · p. 10
  • Discriminative oracles can independently validate candidates, prioritize them for experimental testing, or guide the generator toward more desirable sequences or structures.Source: §3 · p. 10
  • Generative models increase the probability that candidates are valid, while oracles increase the probability that candidates have the desired functionality.Source: §3 · p. 10

4. Conclusions

The review describes protein-design scenarios where deep generative models produced novel proteins, often much faster than traditional approaches, and anticipates further improvements as resources and tools grow.

  • Deep generative models successfully produced novel proteins in multiple protein-design scenarios, often orders of magnitude faster than traditional approaches.Source: §4 · p. 11
  • The authors anticipate further improvements with more available protein sequences and structures, protein-specific machine-learning libraries, and network architectures.Source: §4 · p. 11

6. Conflicts of interest

The disclosure reports PMK's company and advisory-board affiliations.

  • PMK is a cofounder of Resolute Bio Inc. and serves on ProteinQure's scientific advisory board.Source: §6 · p. 11

Appendix A.1. Autoregressive models

Autoregressive models operate on token sequences and use causal or masked language-modeling objectives; these objectives suit different sequence-generation and optimization uses.

  • Autoregressive models operate on token sequences and are typically trained with causal or masked language-modeling objectives.Source: Appendix A.1 · p. 11
  • Causal language modeling predicts each amino acid from preceding amino acids, making it particularly suited to generating novel protein sequences.Source: Appendix A.1 · p. 11
  • Masked language modeling predicts randomly masked amino acids; its bidirectional context suits regional optimization and residue representations, and it can also generate full sequences.Source: Appendix A.1 · p. 11

Appendix A.2. Variational autoencoders (VAEs)

VAEs encode inputs as distributions over latent variables and decode samples from those distributions into reconstructions. They are trained using the negative evidence lower bound (ELBO), rather than directly minimizing the intractable negative marginal probability.

Appendix A.2. Variational autoencoders (VAEs)

  • VAEs encode an input as parameters of a latent-variable distribution and decode a sample from it into a reconstructed output.The latent variables are modeled as independent Gaussians.Source: Appendix A.2 · p. 12
  • Directly minimizing the negative marginal probability is generally intractable because it requires integration over the latent space.The marginal probability is the integral of the decoder likelihood over the latent prior.Source: Appendix A.2 · p. 12
  • VAEs are instead trained by minimizing the negative evidence lower bound (ELBO) of the data given the model parameters.The ELBO includes reconstruction loss and a Kullback–Leibler distance term.Source: Appendix A.2 · p. 12
  • The reconstruction term measures the difference between input and output, while the Kullback–Leibler term compares the latent distribution with its prior.Reconstruction loss can use cross-entropy for categorical data or mean squared error for continuous data.Source: Appendix A.2 · p. 12

Appendix A.3. Normalizing flows (NFs)

Normalizing flows use invertible mappings to connect inputs and latent representations, enabling exact data probabilities through the change-of-variables rule. Their architectures must preserve invertibility and tractable Jacobian determinants, motivating specialized transformations and neural ODE approaches.

Appendix A.3. Normalizing flows (NFs)

  • Normalizing flows use an invertible mapping between inputs and latent variables to calculate exact data probabilities using the change-of-variables rule.The probability is obtained from the latent probability and the determinant of the mapping's Jacobian.Source: Appendix A.3 · p. 12
  • A flow can compose invertible submodels with tractable Jacobian determinants, applying the change-of-variables rule repeatedly across intermediate variables.The composed mapping takes z0 through zK to x.Source: Appendix A.3 · pp. 12–13
  • Flow training minimizes the data's negative log-likelihood, but requires an invertible model with a tractable Jacobian determinant.These requirements constrain the neural network architectures that can be used.Source: Appendix A.3 · p. 13
  • Affine transformations on subsets of channels yield triangular Jacobians, while neural ODEs can use Hutchinson’s trace estimator to approximate the Jacobian trace.Both approaches address the tractable-determinant requirement; other approaches remain an active research area.Source: Appendix A.3 · p. 13

Appendix A.4. Energy-based models (EBMs)

Energy-based models assign low energies to probable states and high energies to improbable ones, using different strategies to train and sample them. The section describes contrastive divergence and noise-contrastive estimation, then presents GANs as a generator-discriminator approach to proposing negative examples.

Appendix A.4. Energy-based models (EBMs)

  • Energy-based models assign low energies to probable states and high energies to improbable ones, avoiding direct log-likelihood maximization and partition-function approximation during training.Because energies are unnormalized, models must be prevented from assigning arbitrarily low energies to every input.Source: Appendix A.4 · p. 13
  • Contrastive divergence updates energies to favor real examples over generated fake examples, but repeated sampling is costly and the evolving sample distribution can destabilize training.Fake examples may be generated with Gibbs sampling, stochastic gradient Langevin dynamics, or another MCMC method.Source: Appendix A.4 · pp. 13–14
  • Noise-contrastive estimation distinguishes real examples from negatives sampled from a predefined distribution, whose tractability and representativeness are key challenges.For categorical-variable/context pairs, negatives can be categories absent from the context; learned variable and context embeddings may aid downstream tasks.Source: Appendix A.4 · p. 14

Appendix A.5. Generative adversarial networks (GANs)

  • GANs train a generator and discriminator together through a minimax game, with the generator proposing negative examples and the discriminator distinguishing them from real examples.In the Wasserstein GAN, the discriminator assigns lower values to real examples and higher values to generated ones, while the generator seeks low discriminator values.Source: Appendix A.5 · p. 14
Loading 2109.13754v1…