PaperlayerView exact revision on arXiv

Source-linked AI summary

Structure Language Models for Protein Conformation Generation

Jiarui Lu, Xiaoyin Chen, Stephen Zhewen Lu, Chence Shi, Hongyu Guo, Yoshua Bengio, Jian Tang

arXiv:2410.18403v2q-bio.BMcs.LG

TL;DR

Protein conformational dynamics matter for biological function, but molecular dynamics is costly and static predictors often miss multiple conformations. SLM models discrete latent structure tokens conditioned on sequence and reports state-of-the-art performance with generation orders of magnitude faster than existing generative methods.

  • Problem

    Molecular dynamics is computationally expensive, and static structure predictors often fail to capture proteins’ multiple conformations.

    Source: §1 · p. 1

  • Method

    SLM encodes structures as discrete latent tokens, models them conditionally on sequence with language models, and decodes the tokens into 3D structures.

    Source: §1 · p. 2

  • Results

    SLM reports state-of-the-art performance across conformation-generation scenarios and generation orders of magnitude faster than existing generative methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework opens potential research avenues for protein conformation generation.

    Source: §1 · p. 2

    The current dVAE architecture leaves a balance between structure disentanglement and reconstruction fidelity to improve.

    Source: §6 · p. 10

Abstract

from arXiv · show

Proteins adopt multiple structural conformations to perform their diverse biological functions, and understanding these conformations is crucial for advancing drug discovery. Traditional physics-based simulation methods often struggle with sampling equilibrium conformations and are computationally expensive. Recently, deep generative models have shown promise in generating protein conformations as a more efficient alternative. However, these methods predominantly rely on the diffusion process within a 3D geometric space, which typically centers around the vicinity of metastable states and is often inefficient in terms of runtime. In this paper, we introduce Structure Language Modeling (SLM) as a novel framework for efficient protein conformation generation. Specifically, the protein structures are first encoded into a compact latent space using a discrete variational auto-encoder, followed by conditional language modeling that effectively captures sequence-specific conformation distributions. This enables a more efficient and interpretable exploration of diverse ensemble modes compared to existing methods. Based on this general framework, we instantiate SLM with various popular LM architectures as well as proposing the ESMDiff, a novel BERT-like structure language model fine-tuned from ESM3 with masked diffusion. We verify our approach in various scenarios, including the equilibrium dynamics of BPTI, conformational change pairs, and intrinsically disordered proteins. SLM provides a highly efficient solution, offering a 20-100x speedup than existing methods in generating diverse conformations, shedding light on promising avenues for future research.

1 INTRODUCTION

Protein conformational dynamics matter for biological function, but traditional simulations are costly and static structure predictors often miss multiple conformations. The paper introduces latent-space language modeling as an alternative for generating sequence-specific conformations.

1 INTRODUCTION

  • Molecular dynamics is computationally expensive, while static structure predictors often fail to capture proteins’ multiple conformations.The paper frames conformational diversity as important for protein function and interactions.Source: §1 · p. 1
  • Flow-based approaches face scaling challenges, while 3D diffusion can be computationally intensive and favor local perturbations over remote conformational modes.The cited discussion attributes the latter tendencies to high-dimensional geometric modeling and L2-based training objectives.Source: §1 · pp. 1–2
  • SLM achieves state-of-the-art performance across conformation-generation scenarios while generating samples orders of magnitude faster than existing generative methods.The introduction reports results across multiple scenarios but gives no specific speedup value here.Source: §1 · p. 2
  • SLM encodes structures as distributions over discrete latent tokens, conditionally models those tokens from amino-acid sequence, and decodes them into 3D conformations.The latent representation removes high-frequency structural details and is intended to capture uncertainty in complex conformations.Source: §1 · p. 2
  • ESMDiff is a BERT-like SLM instantiation fine-tuned from ESM3 using masked discrete diffusion.The paper also evaluates SLMs across varied conformation-generation settings against existing methods.Source: §1 · p. 2

2 RELATED WORK

Related work spans protein sequence language models, generative conformation sampling, and quantized structure representations. The paper positions SLM at their intersection by applying language models to generate conformations over quantized protein structures.

2 RELATED WORK

  • A BPTI illustration depicts residue flexibility through distributions over latent tokens, with different tokens encoding local structural patterns.The caption associates the clusters with Shaw et al. (2010).Source: Figure 1 · p. 2
  • Protein language models span sequence representation, de novo sequence generation, and structure-based inverse folding.The cited examples include ESM-series models, ProGen, ProtGPT2, and inverse-folding models.Source: §2 · p. 2
  • Prior generative approaches include normalizing flows and diffusion or flow-matching models trained to sample protein conformations conditionally.The related-work discussion names the Boltzmann generator, EigenFold, Str2Str, AlphaFlow, ConfDiff, and DiG.Source: §2 · pp. 2–3
  • Discrete structure representations have supported search, alignment, structure-token inputs, and latent-space generation.Examples include FoldSeek, SaProt, ProtT5, PVQD, ProSST, and large-vocabulary dVAEs.Source: §2 · p. 3
  • SLM combines language modeling with quantized protein structures to model and efficiently generate conformations.The paper describes this framework as closely related to concurrent research directions.Source: §2 · p. 3

3 PROTEIN CONFORMATION GENERATION WITH LANGUAGE MODELING

The framework models sequence-to-structure generation through an invariant latent representation: it learns a sequence-conditioned distribution over structure tokens and a decoder that reconstructs 3D structures. It then samples tokens conditionally and decodes them into conformations.

3.1 LEARNING THE SEQUENCE-STRUCTURE DISTRIBUTION

  • The model factorizes sequence-conditioned structure generation into a latent-token prior and a structure decoder, with an invariant latent representation.The learned latent space provides an alternative to explicitly modeling structures in 3D space.Source: §3.1 · pp. 3–4
  • Training separates structure-token representation learning from learning the sequence-conditioned prior over those tokens.The first stage learns an encoder and decoder with a fixed uniform prior; the second fixes them and trains the prior by maximum likelihood.Source: §3.1 · p. 4

3.2 STRUCTURE LANGUAGE MODELING

  • At inference, SLM samples latent structure tokens from a learned sequence-conditioned prior, then decodes them into conformations.The decoder is trained with the encoder so sampled tokens align with structural reconstruction.Source: §3.2 · p. 4
  • The latent-token prior can use encoder-decoder or decoder-only language models, both trained autoregressively for sequence-conditioned token generation.The decoder-only approach treats the amino-acid sequence as a prompt; inference follows left-to-right decoding.Source: §3.2 · pp. 4–5

4 ESMDIFF: A MASKED DIFFUSION INSTANTIATION

ESMDiff instantiates Structure Language Modeling with masked discrete diffusion, fine-tuning ESM3 to generate structure tokens conditioned on amino-acid sequences. The section describes how masked tokens are denoised and how a bidirectional encoder implements this process.

4 ESMDIFF: A MASKED DIFFUSION INSTANTIATION

  • ESMDiff fine-tunes ESM3 with masked discrete diffusion, adapting a pretrained BERT-like model for conditional protein-conformation generation.The paper presents this as an instantiation of SLM and a potential route to broader downstream tasks.Source: §4 · p. 5

4.1 CONDITIONAL MASKED DIFFUSION LANGUAGE MODEL

  • Masked diffusion treats [MASK] as an absorbing state: forward corruption masks tokens, while the conditional reverse process gradually unmasks them.The reverse denoiser conditions on amino-acid types; training loss applies to masked latents.Source: §4.1 · pp. 5–6, Figure 4 · p. 6

4.2 BIDIRECTIONAL ENCODER AS DENOISING NETWORK

  • A bidirectional encoder factorizes denoising across token positions, with amino-acid and structure-token positions aligned; unmasked tokens are copied and [MASK] is excluded from outputs.The implementation uses a BERT-style transformer and replaces ESM3’s pretrained LM head with a randomly initialized augmented-vocabulary head.Source: §4.2 · pp. 6–7

5 EXPERIMENTS

The experiments compare structure language models with existing methods across BPTI dynamics, conformational-change pairs, and intrinsically disordered proteins, and assess runtime and structural inpainting. Reported findings include faster sampling, improved matching for a difficult BPTI cluster, and strong MSA-based performance on conformational-change pairs.

5 EXPERIMENTS

  • The experiments use a frozen pretrained dVAE tokenizer, structure-language-model variants, and 112.4k PDB training structures dated on or before May 1, 2020.Benchmark settings cover BPTI dynamics, conformational-change pairs, and IDPs.Source: §5 · pp. 7–8

5.1 STRUCTURAL DYNAMICS OF BPTI

  • SLMs significantly improve matching to BPTI’s difficult remote-folding Cluster 3, with smaller matching RMSD.Table 2 reports best-matching RMSD across five kinetic clusters for ensembles of 100.Source: Table 2 · p. 8, §5.1 · p. 8

5.2 CONFORMATION CHANGING PAIRS

  • MSA-based methods generally outperform other model families on fold-switching and apo/holo conformational-change pairs.The authors highlight the importance of MSA for generating stable conformationally changing protein targets.Source: Table 3 · p. 8, §5.2 · p. 9

5.3 INTRINSICALLY DISORDERED PROTEINS

  • The IDP benchmark contains 114 experimentally validated ensembles; performance is evaluated by MAE for pairwise distance, radius of gyration, and contact map.The study excludes records similar to the training set and does not use alignment-based metrics such as TM-score for IDPs.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 RUNTIME ANALYSIS

  • 20–100× faster than diffusion baselines such as AlphaFlow, SLMs scale better with protein size when sampling ensembles of 100.Runtime was measured as elapsed wall-clock time; Figure 5 profiles the methods.Source: Figure 5 · p. 9, §5.4 · p. 10

5.5 CASE STUDY: STRUCTURAL INPAINTING OF NANOBODY

  • ∼5.3 ± 0.3 seconds on one A100 generated 20 nanobody inpaintings whose loops resembled native NMR conformations.The case study uses round-trip masked diffusion to sample the complementarity-determining regions of a Llama nanobody.Source: Figure 6 · p. 10, §5.5 · p. 10

6 CONCLUSION AND LIMITATIONS

SLMs generate conformations by conditionally sampling latent structure tokens and decoding them into structures; the paper reports benchmarking across multiple conformation-generation tasks. The conclusion also identifies limitations and future directions, while appendices document baselines, runtime profiling, and reproducibility.

6 CONCLUSION AND LIMITATIONS

  • SLMs demonstrate compelling performance and application potential across various conformation-generation tasks.Inference has two stages: conditional sampling of latent structure tokens followed by roto-translation-invariant structure decoding.Source: §6 · p. 10
  • The current study identifies dVAE reconstruction fidelity versus structure disentanglement, latent-space choice, model inductive biases, atomic modalities, and MSA conditioning as areas for future work.The authors suggest continuous latents, an atomic structural auto-encoder for side chains and ligand or RNA, and MSA-conditioned SLMs.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

  • The study makes its source training and inference code publicly available and points to appendices for implementation, sampling, evaluation, and theoretical details.Appendix A covers implementation and training, Appendix B describes ESM3 and ESMDiff sampling, Appendix D covers evaluation, and Appendix G.2 presents theoretical derivations or proofs.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 BASELINE EVALUATIONS

  • Baseline evaluations compare SLMs with EigenFold, Str2Str, MSA subsampling, AlphaFlow, ESMFlow, and idpGAN.The appendix specifies inference settings for these methods, including sampling and MSA configurations.Source: §A.1 · p. 16

A.2 RUNTIME PROFILING

  • Runtime profiling measures wall-clock time for ensembles of 100 structures across protein lengths.For MSA-based methods, the MSA search is performed once and amortized across batches; S-T5 and S-GPT cache key-value pairs during inference.Source: §A.2 · p. 16

A.3 TRAINING DETAILS

The appendix records training choices for S-T5, S-GPT, and masked-diffusion fine-tuning of ESM3, and describes how the IDP test set was curated.

A.3 TRAINING DETAILS

  • S-T5 has 384M trainable parameters, trains for up to 30 epochs, and uses the best checkpoint by held-out structure-token NLL.Precomputed ESM3 sequence embeddings are fed directly to the transformer encoder, bypassing its input-token embedding layer.Source: §A.3 · p. 16, Table S1 · p. 17
  • S-GPT has 961M trainable parameters and inserts a learnable, randomly initialized [sep] token between sequence and structure tokens.Its loss uses λ to balance sequence-token and structure-token NLL; the appendix gives training settings and reports the best checkpoint.Source: §A.3 · p. 16, Table S2 · p. 17
  • ESM3 fine-tuning uses a log-linear masked-diffusion schedule and replaces the structure-track LM head with a randomly initialized head for an augmented 4101-token vocabulary.The training algorithm samples a timestep and masking probability, masks structure tokens, computes the negative ELBO, and updates model parameters.Source: §A.3 · pp. 16–17

A.4 CURATION OF IDP TEST SET

  • The IDP evaluation set was filtered from PED entries by sequence length, ensemble size, sequence similarity, and experimental validation.The final selection excludes records with more than 90% sequence identity to PDB training data and computationally generated ensembles.Source: §A.4 · pp. 17–18

A.5 TRAINING DATA OF BASELINE MODELS

This appendix describes the baseline training-data reference, zero-shot ESM3 sampling, and ranking strategies for iterative decoding, including a protein-specific secondary-structure schedule.

A.5 TRAINING DATA OF BASELINE MODELS

  • The appendix points to a table listing training data and base models for the methods in the study.The supplied table caption identifies its scope but does not provide the underlying entries.Source: §A.5 · p. 18, Table S3 · p. 18

B.1 ZERO-SHOT CONFORMATION GENERATION OF ESM3

  • Zero-shot conformation sampling uses the open-source 1.4B ESM3 model with sequence and structure tracks active and other tracks set to default mask values.Sampling uses 25 steps, temperature 1.0, top-p 0.95, and entropy-ranked iterative decoding for the reported experiments.Source: §B.1 · p. 18

B.2 POSITION-RANKED ITERATIVE DECODING

  • Iterative decoding ranks masked positions, selects positions to unmask at each step, samples their structure tokens, and updates the masked set.The schedule divides the protein length across decoding steps; positions are chosen using a ranking function and the model's conditional token distribution.Source: §B.2 · pp. 18–19
  • Entropy ranking decodes positions with smaller predictive entropy first, recalculating predictions from the unmasked tokens at the preceding step.The appendix also considers maxlogit ranking, which prioritizes positions with the highest predicted logit.Source: §B.2 · p. 19
  • Secondary-structure ranking orders masked positions from predicted structured regions toward more disordered regions, using ESM3's SS8 prediction heads.SS8 labels map helices, strands, bends, and coils to a finite ordering score; the authors' rationale is to predict structural regions before loops.Source: §B.2 · p. 19

B.3 INFERENCE OF CONDITIONAL MASKED DIFFUSION MODEL

ESMDiff supports conditional generation through iterative decoding or DDPM sampling, and masked diffusion also supports inpainting selected structural regions. The appendix notes choices in DDPM sampling and explains how the inpainting process preserves unselected regions.

B.3 INFERENCE OF CONDITIONAL MASKED DIFFUSION MODEL

  • After fine-tuning, ESMDiff supports conditional sequence-to-structure generation through iterative decoding or vanilla DDPM ancestral sampling.The DDPM procedure progressively unmasks tokens over 25 steps; iterative decoding can use the fine-tuned ESM3 backbone without changes.Source: §B.3 · p. 19
  • DDPM sampling uses temperature 1.0 and does not apply the positional ranking function; the authors identify both controls as future work.The described DDPM sampling uses the time-conditioned fine-tuned model.Source: §B.3 · p. 20

B.4 COMPARISON BETWEEN ESM AND ESMDIFF

  • ESMDiff fully conditions on amino acid types and trains structure tokens with masked diffusion; masking ratios should reflect downstream inference inputs.For conformation generation, the authors reason that little or no sequence masking and scheduled structure masking are beneficial.Source: §B.4 · p. 20

C DETAILS OF CONFORMATION INPAINTING

  • Inpainting masks only the selected latent positions, reverses diffusion, and preserves copied unmasked positions, restricting sampling to the target region.The example fixes a nanobody framework while sampling its three CDR loops; the NMR and generated ensembles each contain 20 conformations.Source: C DETAILS OF CONFORMATION INPAINTING · p. 21

D EVALUATION METRICS

The evaluation uses complementary measures of ensemble distribution similarity, structural coverage and fidelity, flexibility, and validity. For IDPs, it also evaluates ensemble-average feature error; fine-tuning choices have task-dependent effects.

D EVALUATION METRICS

  • JS divergence compares reference and generated ensemble distributions using pairwise distances, slow TICA components, and radius of gyration, with only Cα atoms used for compatibility with baselines.Pairwise-distance histograms use 50 bins; TICA uses the first two components fitted on reference MD trajectories.Source: D EVALUATION METRICS · p. 22
  • Validity is the fraction of generated conformations without steric clashes, with clashes defined using a 3.0 Å distance threshold.The threshold applies to atom pairs as specified in the metric definition.Source: D EVALUATION METRICS · p. 22
  • TM-ens measures reference-state coverage by averaging each state's best generated-sample TM-score, while RMSD-ens averages minimum RMSD values across reference conformations.Both scores assess structural similarity under limited sampling; higher TM-ens indicates better coverage.Source: D EVALUATION METRICS · p. 22
  • Residue flexibility is evaluated by Pearson correlation for global and per-target ResFlex.The supplied passage defines ResFlex as an absolute deviation from the center, but does not specify the remainder of that definition.Source: D EVALUATION METRICS · p. 22
  • For IDPs, MAE compares reference and generated ensemble-average features, while contact maps represent contact probabilities using an 8.0 Å distance cutoff.Mean and median MAE are reported across 114 targets; contact frequencies are normalized over the ensemble.Source: D EVALUATION METRICS · p. 23

E.1 FINE-TUNING CONFIGURATION

  • Adding sequence-track dropout, masking, or prediction to ESMDiff fine-tuning improves BPTI results but decreases accuracy on apo/holo and fold-switching evaluation.The reported configuration uses 25% track dropout, 10% masking, and equal sequence- and structure-prediction loss weights.Source: §E.1 · p. 23

E.2 CHANGING OF TEMPERATURES

Sampling temperature and step count affect validity and diversity differently across SLMs. The discussion also identifies constraints in reconstructing short-timescale MD ensembles and criteria for useful structure-token quantization.

E.2 CHANGING OF TEMPERATURES

  • At T ≤1.0, S-GPT maintains high validity while achieving greater diversity; above T=1.5, S-GPT and S-T5 validity deteriorate, whereas ESM3 and ESMDiff remain valid but gain little diversity at higher temperatures.The temperature sweep evaluates BPTI ensembles of N=100 using validity and TM-diversity.Source: §E.2 · pp. 23–24, Figure S1 · p. 24

E.3 CHANGING OF SAMPLING STEPS

  • Increasing sampling steps benefits ID sampling, but accuracy for the other two methods plateaus after 10 steps and diversity changes little; DDPM has the highest validity at nearly all step counts except 100.The tested range is 5–100 steps, with computational cost growing linearly in K.Source: §E.3 · p. 24, Figure S2 · p. 24

E.4 EVALUATING ON THE ATLAS MD ENSEMBLES

  • ESM3-based models struggled to reconstruct ATLAS test ensembles, whose 100 ns simulations may miss slower conformational changes that challenge categorical structure tokens.The authors identify learning from MD ensembles as an area for further work.Source: §E.4 · p. 25

F.1 QUANTIZED REPRESENTATION OF STRUCTURES

  • Useful quantization should encode local structure patterns with limited interactions between tokens, remain invariant to global rotation and translation, and cover structural variation without an excessively large vocabulary.The discussion links disentanglement to efficient language modeling and interpretability, and warns that oversized vocabularies can encode noise and hinder generalization.Source: §F.1 · p. 26

F.2 RECONSTRUCTION ON TEST DATASETS

The reconstruction check finds dataset-dependent limitations in dVAE structure encoding, with weaker accuracy for fold-switching pairs than for apo/holo pairs.

F.2 RECONSTRUCTION ON TEST DATASETS

  • A non-negligible fraction of fold-switching and IDP targets has reconstruction RMSD above 0.5 Å after dVAE quantization.Figure S3 summarizes target RMSD distributions across the participating test sets.Source: §F.2 · p. 26, Figure S3 · p. 27
  • The dVAE reconstructs apo/holo pair structures more accurately than fold-switching structures, which may help explain SLMs' relatively inferior fold-switching performance.Figure S4 shows RMSD histograms for the two pair types.Source: §F.2 · p. 26, Figure S4 · p. 27
  • Here, “disentangle” means local-to-global organization with restricted token receptive fields, not semantic-level structural description.The text cautions that conditional independence of latent tokens given structure does not necessarily imply that relation given sequence.Source: §F.2 · p. 26

G EXTENDED METHODOLOGY DISCUSSION

The appendix describes discrete diffusion as interpolation over categorical structure tokens, derives conditional masked diffusion training, and illustrates generated ensembles across BPTI, conformational-change pairs, and IDPs.

G.1 REVISITING DISCRETE DIFFUSION AS DISTRIBUTION INTERPOLATION

  • With an explicit stationary distribution, discrete diffusion interpolates between the current categorical state and that distribution as α(t) decreases; sequence and structure tokens align by position.The passage describes the continuous-time formulation and a fixed-length context for sequence-conditioned generation.Source: §G.1 · p. 28, §G.2.1 · p. 28

G.2.1 DERIVATION OF THE EVIDENCE LOWER BOUND IN EQUATION 1; G.2.2 DERIVATIONS OF MARGINAL AND POSTERIOR IN G.1

  • The ELBO derivation treats latent tokens as an approximate posterior given structure and assumes structure is conditionally independent of sequence given those tokens; the forward marginal mixes a token state with the stationary prior.The forward marginal is expressed as a categorical distribution with interpolation weight μ(s,t).Source: §G.2.1 · p. 29, §G.2.2 · p. 29

G.2.3 DERIVATION OF CONDITIONAL MASKED DIFFUSION IN 4.1

  • Conditional masked diffusion uses the [MASK] token as its stationary distribution and derives a posterior for the previous token from the current token and original token.The posterior expression distinguishes whether the current token is masked.Source: §G.2.3 · pp. 30–31

G.2.4 LEARNING OBJECTIVE FOR CONDITIONAL MASKED DIFFUSION

  • For masked diffusion, the ELBO reduces to a multi-step diffusion loss because reconstruction and prior terms cancel; training can estimate the time integral by sampling t uniformly.The ELBO contribution is nonzero only when the current token is masked.Source: §G.2.4 · pp. 31–32

H STRUCTURE GALLERY

  • The structure gallery presents ESMDiff (DDPM sampling) ensembles alongside reference structures or ground-truth ensembles across several protein scenarios.The gallery includes ensemble and contact-map visual comparisons.Source: H STRUCTURE GALLERY · p. 32

H.1 PROTEIN BPTI

  • For BPTI, the gallery shows best-matched samples from an N=100 modeled ensemble for each of five kinetic clusters.The samples are structurally aligned with the reference structure.Source: §H.1 · p. 32

H.2 CONFORMATION CHANGE PAIRS

  • For apo/holo conformation-changing pairs, Figure S6 overlays the sampled ensemble with the ground-truth structures.The sampled conformations are shown in khaki and the reference structures in transparent rainbow colors.Source: §H.2 · p. 32, Figure S6 · p. 33

H.3 INTRINSICALLY DISORDERED PROTEIN

  • In the IDP gallery, PED00433 is a failure case: the generated ensemble has reduced diversity at both terminal fragments relative to the experimental ensemble.The figure also compares sampled and ground-truth ensembles and their contact maps.Source: §H.3 · p. 33, Figure S7 · p. 34

Related papers

Loading 2410.18403v2…