Source-linked AI summary
Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Models
Namrata Anand, Tudor Achim
TL;DR
Designing proteins with specified structures and chemical properties requires generative models that handle protein geometry, sequence, and scale. The paper introduces an experimental-data-trained equivariant diffusion model conditioned on compact topology specifications, and reports physically plausible, varied proteins with controllable design capabilities across PDB domain types.
Problem
Protein design seeks structures and sequences with researcher-specified functional properties, but protein generation lacks image-like representations and canonical orientations.
Method
The model uses equivariant denoising diffusion for structure, sequence, and rotamer generation, conditioning samples on compact protein-topology constraints.
Results
The model produces physically plausible and highly varied large protein domains across PDB domain types, with demonstrated loop, sequence, rotamer, and topology design capabilities.
Takeaways & Limitations
Compact constraints make conditional sampling useful for modifying existing proteins and exploring end-to-end protein design.
Takeaways & Limitations
Current quality may improve by cross-conditioning structure, sequence, and rotamers instead of generating them sequentially.
Abstract
from arXiv · showhide
Proteins are macromolecules that mediate a significant fraction of the cellular processes that underlie life. An important task in bioengineering is designing proteins with specific 3D structures and chemical properties which enable targeted functions. To this end, we introduce a generative model of both protein structure and sequence that can operate at significantly larger scales than previous molecular generative modeling approaches. The model is learned entirely from experimental data and conditions its generation on a compact specification of protein topology to produce a full-atom backbone configuration as well as sequence and side-chain predictions. We demonstrate the quality of the model via qualitative and quantitative analysis of its samples. Videos of sampling trajectories are available at https://nanand2.github.io/proteins .
1 Introduction
The paper addresses protein structure generation, where existing data-driven methods face representation and rotational-orientation challenges. It introduces an equivariant diffusion model that generates varied, plausible large proteins across experimentally characterized domain topologies from compact constraints.
- Motivation: Protein structure generation remains difficult because proteins lack both an image-like discretized representation and a canonical orientation.Distance-matrix modeling can produce inconsistent predictions, while non-rotationally invariant methods must account for arbitrary orientations.
- Prior limitations: Existing methods have been limited either to small molecules or to large proteins in highly restricted single-domain-topology settings.
- Illustration: Figure 1 illustrates four unconstrained samples per scenario, showing variability and hydrogen-bond patterns, while comparing geometric distributions with native crystal structures.
- Contribution: The paper introduces a fully data-driven diffusion model for protein structure, sequence, and rotamers across the full range of PDB domain types.The model uses equivariance through invariant point attention and handles rotational and discrete sequence variables during generation.
- Contribution: The model conditions generation on a compact specification of protein topology to produce plausible large protein structures and sequences.
2 Approach
The approach represents proteins through coordinates, local frames, residues, and side-chain torsions, then learns their generation through iterative denoising. Equivariant attention, manifold-aware interpolation, rotationally invariant losses, masked sequence diffusion, and compact topology constraints support conditional sampling.
- Protein representation: The model learns distributions over Cα coordinates, canonical-frame rotations, amino-acid identities, and side-chain torsion angles.These variables jointly describe backbone geometry, sequence, and rotamer configurations.
- Diffusion framework: Diffusion models generate proteins by learning a reverse denoising process that maps samples from a random prior toward the data distribution.
- Equivariant diffusion: Rotations and torsion angles are diffused by interpolation on their manifolds rather than by Euclidean perturbation.Uniform rotations are interpolated with SLERP on SU(2), while rotamer angles use interpolation on the unit circle.
- Equivariance: Invariant point attention processes local-frame features so denoising remains equivariant to protein rotations and translations.FAPE aligns predicted and ground-truth local frames, making the training loss invariant to dataset orientation.
- Sequence diffusion: Sequence generation uses progressively masked residues during training and reverses masking from all residues masked toward the unmasked sequence.
- Conditional sampling: A compact topology specification encodes secondary-structure blocks and their adjacencies, which condition attention updates and sampled structures.The conditioning information includes relationships between helices, beta sheets, and loops.
3 Experiments
Experiments evaluate separate diffusion models for structure, sequence, and rotamers, then test inpainting, controllable generation, and joint structure–sequence modeling. The samples are physically plausible, match protein biophysical distributions, support novel structural completion and modification, and achieve sequence-design and rotamer-packing performance comparable to baselines.
- Experimental setup: The models use separate diffusion processes for structure, sequence, and rotamers, with sequence and rotamer models trained on progressively richer structural information.Structure diffuses x_Cα and q, sequence diffuses r, and rotamers diffuse χ; a separate joint model is also evaluated.
- 3.1 Context-free Generation: Test-set generation produces highly variable, physically plausible protein structures with intra-backbone hydrogen bonds in helices and beta sheets.Evaluation spans four unseen CATH-topology test backbones covering all-alpha, alpha–beta, and all-beta classes.
- 3.1 Context-free Generation: Generated bond lengths and angles show good histogram overlap with native structures, while backbone torsions are consistent with the Ramachandran distribution.These results indicate that the model learned biophysical priors directly from the data distribution.
- 3.2 Inpainting and Controllable Generation: Inpainting preserves fixed residues while producing completed regions whose bond geometries match native distributions and whose loop samples exhibit discrete atomic-level modes.The additional model conditions on existing structures by diffusing only the complement of fixed regions.
- 3.2 Inpainting and Controllable Generation: Controllable sampling modifies protein designs through idealized topology generation, loop-length engineering, and secondary-structure-length modification.The resulting synthetic structures are distinct from natural PDB structures while remaining physically plausible.
- 3.3 Sequence Design and Rotamer Packing: Sequence recovery and rotamer packing are comparable to reported baselines, while the approach is faster by an order of magnitude.Rotamer performance remains comparable at the most stringent 5- and 10-degree metric cutoffs.
- 3.4 Joint Modeling: Joint backbone–sequence inpainting can at times nearly recover native sequences and backbone atom positions, and supports jointly generating loop backbones and candidate sequences.The authors demonstrate this capability on variable antibody CDR loops over a fixed immunoglobulin backbone.
- 3.4 Joint Modeling: The current joint model generates structure, sequence, and rotamers sequentially, while future cross-conditioning is expected to improve sample quality.The paper leaves models with reciprocal conditioning among all three components for future work.
4 Conclusions
The model produces physically plausible, varied protein domains and supports joint structure–sequence modification, including loop design and topology variation. The paper also identifies future applications and risks associated with powerful protein-design tools.
- The model generates physically plausible and highly varied protein structures, sequences, and rotamers across protein domain types in the PDB.
- The compact constraint specification supports modifying existing proteins by designing loops and varying underlying topology.
- Future directions include replacing AlphaFold2 recycling, guiding sampling with auxiliary energy functions, and applying the model to rational design and structure determination.
- Joint contextual inpainting can nearly recover native backbone positions and sequences in some masked regions, enabling full-atom loop generation.
- Powerful protein-design tools may accelerate progress faster than the research community can self-regulate, creating risks for therapeutic and vaccine design workflows.
A.1 Training
The models are trained with AdamW and cosine learning-rate decay, using single K80 or V100 GPUs and gradient accumulation for larger effective batches.
- Training uses AdamW with cosine learning-rate decay, single K80 or V100 GPUs, and gradient accumulation to increase effective batch sizes.
A.2 Constraint embedding
Constraint embeddings encode adjacency relationships between secondary-structure elements using distance thresholds and beta-strand orientation. The embedding network combines a transformer with triangle-attention blocks.
- Beta strands are adjacent below 5 Å minimum Cα distance, while beta–helix and helix–helix pairs use a 7 Å threshold; beta-pair orientation is also encoded.
- The constraint embedding network contains a 5-layer GPT-3-like transformer followed by 25 triangle-attention layers.
A.3 Diffusion decoders
Separate diffusion decoders handle structure, sequence, rotamers, and inpainting, with structure updates driven by invariant point attention and sequence or rotamer models operating without forward-pass backbone updates.
- Structure diffusion: The structure diffusion decoder conditions on constraints and noised structures, predicts corrected backbones, and updates rotations and translations through invariant point attention.
- Structure diffusion: The structure diffusion network has 12 layers and is trained on subsampled backbones up to 256 residues with T = 1000.
- Sequence diffusion: Sequence diffusion uses no constraint embeddings and performs no backbone updates during the forward pass.
- Sequence diffusion: The sequence network has 15 layers and is trained on subsampled backbones up to 128 residues with T = 100.
- Rotamer diffusion: Rotamer diffusion takes full-atom proteins with diffused side chains and performs no backbone updates or constraint embedding during the forward pass.
- Structure inpainting: Structure inpainting keeps the structure fixed except for masked regions and trains with block or contiguous diffusion masking schemes.
- Joint structure and sequence inpainting: The structure inpainting model is fine-tuned from structure diffusion, while joint inpainting freezes that model and fine-tunes sequence diffusion on predicted structures.
B.1 TIM-barrel topology sampling
The model decodes TIM-barrel structures from sampled secondary-structure lengths and repeated four-fold symmetric topology specifications. Structures longer than 256 residues are excluded.
- TIM-barrel designs use a parallel beta-barrel surrounded by helices adjacent to their immediate neighbors.
- Each TIM-barrel quarter follows the block order L-E-L-H-L-E-L-H-L, with L denoting loop, E beta strand, and H helix.
- Randomly sampled secondary-structure lengths are repeated 4× to create four-fold symmetric topologies.
- Topologies longer than 256 residues are removed before diffusion decodes the sampled secondary-structure strings and block adjacencies.
B.2 Ig domain joint loop backbone and sequence sampling
The model samples modified Ig-domain loop regions while preserving the remaining structure, then packs sequence side-chains for the completed design. The section also presents random backbone and structure-modification samples.
- An Ig domain is modified by masking CDR loops and an additional potentially interacting loop, then sampling random lengths for those regions.
- The structure’s adjacency information remains unchanged because loop regions are not encoded with adjacency information.
- The section includes random generated backbone samples and random generated inpainting samples as qualitative outputs.
- Structure-modification examples include random TIM-barrels with varied underlying four-fold symmetric topologies and sampling loops of varying lengths.