PaperlayerView exact revision on arXiv

Source-linked AI summary

DPLM-2: A Multimodal Diffusion Protein Language Model

Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, Quanquan Gu

arXiv:2410.13782v1cs.LGq-bio.QM

TL;DR

Protein generation must account for both sequence and structure, yet existing approaches often model the modalities separately. DPLM-2 extends a discrete diffusion protein language model to both, and the authors report competitive co-generation alongside performance across conditional generation tasks.

  • Problem

    Many protein-generation approaches model sequence or structure separately, limiting joint modeling of their relationship despite the need to understand and generate both modalities.

    Source: §1 · pp. 1–2

  • Method

    DPLM-2 extends discrete diffusion DPLM to sequence and structure, using discrete structure tokens and training on experimental and high-quality synthetic structures.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    DPLM-2 achieves competitive co-generation versus structure-based approaches and supports folding, inverse folding, multimodal motif scaffolding, and structure-aware predictive representations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors report that a multimodal protein model can be built with moderate high-quality data, a structure tokenizer, and publicly accessible sequence-pretrained language models.

    Source: §1 · p. 3

    Limited structure data constrain DPLM-2's ability to learn robust representations; longer protein chains and multimers remain future-study considerations.

    Source: §5 · p. 11

Abstract

from arXiv · show

Proteins are essential macromolecules defined by their amino acid sequences, which determine their three-dimensional structures and, consequently, their functions in all living organisms. Therefore, generative protein modeling necessitates a multimodal approach to simultaneously model, understand, and generate both sequences and structures. However, existing methods typically use separate models for each modality, limiting their ability to capture the intricate relationships between sequence and structure. This results in suboptimal performance in tasks that requires joint understanding and generation of both modalities. In this paper, we introduce DPLM-2, a multimodal protein foundation model that extends discrete diffusion protein language model (DPLM) to accommodate both sequences and structures. To enable structural learning with the language model, 3D coordinates are converted to discrete tokens using a lookup-free quantization-based tokenizer. By training on both experimental and high-quality synthetic structures, DPLM-2 learns the joint distribution of sequence and structure, as well as their marginals and conditionals. We also implement an efficient warm-up strategy to exploit the connection between large-scale evolutionary data and structural inductive biases from pre-trained sequence-based protein language models. Empirical evaluation shows that DPLM-2 can simultaneously generate highly compatible amino acid sequences and their corresponding 3D structures eliminating the need for a two-stage generation approach. Moreover, DPLM-2 demonstrates competitive performance in various conditional generation tasks, including folding, inverse folding, and scaffolding with multimodal motif inputs, as well as providing structure-aware representations for predictive tasks.

1 INTRODUCTION

DPLM-2 extends a discrete diffusion protein language model to jointly model protein sequences and structures. It targets multimodal generation and understanding with a structure tokenizer and a warm-up strategy that draws on pretrained sequence models.

1 INTRODUCTION

  • Most prior approaches use separate models for sequence and structure, while multimodal protein modeling must address continuous structural data and limited structure-data scale.The introduction also notes that Multiflow underperforms at co-generating structurally compatible sequences and falls short on folding given sequences.Source: §1 · pp. 1–2
  • DPLM-2 extends discrete diffusion DPLM to both modalities, using lookup-free quantization for structures, evolutionary-data warm-up, and self-mixup training for sequence learning.It is trained on experimental and high-quality synthetic structures to learn joint, marginal, and conditional sequence-structure distributions.Source: §1 · p. 3
  • The authors report that multimodal modeling can be effective with moderate high-quality data, a structure tokenizer, and publicly accessible sequence-pretrained language models.DPLM-2 builds on sequence-based DPLM rather than relying only on structural training data.Source: §1 · p. 3
  • DPLM-2 achieves competitive sequence-structure co-generation versus structure-based approaches, with better alignment to natural proteins in secondary-structure statistics.The authors report designable and diverse proteins whose sequences and structures are consistent.Source: §1 · p. 3
  • Its multimodal design supports sequence-conditioned folding, structure-conditioned inverse folding, motif scaffolding with multimodal inputs, and structure-aware representations for predictive tasks.The introduction describes motif scaffolding as more successful and reports additional benefits for a range of predictive tasks.Source: §1 · p. 3

2 PRELIMINARIES

Protein modeling treats each residue as paired sequence and coordinate modalities, so generative tasks can be framed by their conditioning and output across these modalities. DPLM uses discrete diffusion to learn and generate sequences by iterative denoising.

2.1 GENERATIVE MODELING FOR PROTEIN

  • Protein residues are represented by amino-acid identity and three-dimensional coordinates, and generative tasks specify conditioning and output across sequence and structure modalities.These tasks include folding, inverse folding, sequence or structure generation, and sequence-structure co-generation.Source: §2.1 · pp. 3–4, Table 1 · p. 3

2.2 DIFFUSION PROTEIN LANGUAGE MODEL (DPLM)

  • DPLM uses absorbing discrete diffusion to perturb sequences toward an all-mask state and learn a reverse process that denoises toward the data distribution.Its learning objective can be simplified to reweighted cross-entropies, resembling masked language modeling at arbitrary noise levels.Source: §2.2 · p. 4
  • At inference, DPLM iteratively predicts clean sequence tokens and samples progressively less noisy sequences; its sequence representations can also support predictive tasks.The reverse process is described as an iterative mask-predict approach.Source: §2.2 · p. 4

3 DPLM-2: A MULTIMODAL DIFFUSION PROTEIN LANGUAGE MODEL

DPLM-2 extends a sequence-based diffusion protein language model to jointly model structure and sequence using discrete structure tokens. Its design combines multimodal denoising, warm-up from pretrained DPLM, and lookup-free structure quantization.

3.1 OVERVIEW

  • DPLM-2 concatenates residue-aligned structure and sequence tokens and jointly denoises them, enabling co-generation without a cascaded generation paradigm.Separate noise schedules for the two modalities permit sampling from their marginals and conditionals.Source: §3.1 · p. 5

3.2 EFFICIENT WARM-UP FROM PRE-TRAINED SEQUENCE-BASED DPLM

  • Warm-up from pretrained sequence-based DPLM transfers evolutionary information to structure modeling while LoRA limits deviation from original parameters and reduces training costs.The structure dataset is much smaller than the UniRef50 sequence database (200K versus 45M).Source: §3.2 · p. 5

3.3 LEARNING STRUCTURE TOKENIZATION

  • The structure tokenizer encodes backbone coordinates into invariant features, quantizes them into discrete residue tokens, and decodes tokens to reconstructed coordinates.The model uses a GVP-based encoder and an IPA-based decoder, and adopts LFQ after conventional VQ-VAE struggled to train.Source: §3.3 · p. 6
  • LFQ significantly outperforms VQ-VAE in reconstruction accuracy and trains in 2 rather than 15 days on 8 A100s.A codebook size of 8192 gives the best compression-reconstruction trade-off; the tokens correlate with secondary structures.Source: §3.3 · p. 6, Figure 2 · p. 6

4 EXPERIMENTS

The experiments evaluate DPLM-2 across unconditional generation, folding, inverse folding, motif scaffolding, and predictive tasks. Results report plausible and diverse co-generation, natural-like secondary-structure proportions, and conditional-generation and representation-learning performance, alongside limitations in some settings.

4.1.1 DPLM-2 ENABLES HIGH-QUALITY, DIVERSE AND NOVEL PROTEIN SEQUENCE AND STRUCTURE GENERATION

  • For proteins longer than the 512-residue training cutoff, DPLM-2's generated sequences have pLDDT scores close to DPLM's.The paper also presents generated samples up to 700 residues and symmetric-oligomer designs made by duplicating predicted structure and sequence tokens.Source: §4.1.1 · pp. 7–8
  • DPLM-2 co-generation achieves high designability across lengths, with most sc-TM values above 0.9, and is comparable to Multiflow; retrained Multiflow lags behind.The paper also reports that co-generation has the highest scTM among DPLM-2's generation modes. For lengths beyond its 512-residue training cutoff, sampled sequences have pLDDT scores close to DPLM's.Source: §4.1.1 · p. 7, Table 2 · p. 8

4.1.2 DPLM-2 GENERATES PROTEINS THAT RESEMBLES NATURAL PROTEINS

  • DPLM-2 produces the most natural-like secondary-structure proportions among the compared methods, while its proteins span a wider distribution than Multiflow's helix-rich samples.Higher loop proportions increase scRMSD, which the authors relate to the flexibility of long loops that DPLM-2 often generates.Source: Figure 4 · p. 8, §4.1.2 · pp. 8–9

4.1.3 ABLATION STUDY

  • Sequence pre-training and data augmentation improve designability and diversity, especially for proteins longer than 300 residues, and combining them gives the best overall performance.The authors suggest that the limited number of long PDB proteins leaves training insufficient; data augmentation is less effective than sequence pre-training for diversity.Source: Table 3 · p. 9, §4.1.3 · p. 9

4.2 FORWARD FOLDING (SEQUENCE-CONDITIONED STRUCTURE PREDICTION)

  • Forward folding is evaluated as sequence-conditioned structure prediction against baselines on the CAMEO 2022 datasets.The supplied results identify the comparison and dataset but do not report its outcome.Source: Table 4 · p. 9

4.3 INVERSE FOLDING (STRUCTURE-CONDITIONED SEQUENCE GENERATION)

  • Inverse folding is evaluated using amino-acid recovery and the self-consistency TM-score between the native structure and the ESMFold-predicted structure.The supplied passages define the evaluation but do not report comparative outcomes.Source: §4.3 · p. 10, Table 5 · p. 10

4.4 SCAFFOLDING WITH MIXED-MODAL MOTIF CONDITIONING

  • For motif scaffolding, DPLM-2 outperforms or matches corresponding approaches in most cases and performs better than ESM3 in simultaneous scaffold sequence-and-structure generation.The authors note that these results have not been experimentally verified.Source: Figure 5 · p. 10, §4.4 · p. 10

4.5 EVALUATION OF PROTEIN REPRESENTATION LEARNING

  • DPLM-2 improves on sequence-only methods on some predictive tasks, but trails SaProt on most tasks and DPLM on certain tasks.The authors hypothesize that limited, differing structure-training data may contribute to catastrophic forgetting and weaker representations.Source: Table 6 · p. 10, §4.5 · p. 11, Table 7 · p. 11

5 DISCUSSIONS

The discussion identifies constraints in structure data and discrete structure tokens, and points to longer chains, multimers, and data-space generation as future directions.

5 DISCUSSIONS

  • Limited structure data constrains DPLM-2's ability to learn robust representations.The authors identify longer protein chains and multimers as areas for future study.Source: §5 · p. 11
  • Discrete structure tokens facilitate multimodal modeling and co-generation but may lose fine-grained structural detail and control.The named examples are precise atomic positions and inter-atomic distances.Source: §5 · p. 11
  • The authors propose integrating strengths of data-space structure-based generative models into sequence-based multimodal language models.Source: §5 · p. 11

A.1 TACKLING EXPOSURE BIAS IN DISCRETE DIFFUSION WITH SELF-MIXUP TRAINING STRATEGY

The section frames exposure bias as a training-inference mismatch in discrete diffusion and describes self-mixup training as a response.

A.1 TACKLING EXPOSURE BIAS IN DISCRETE DIFFUSION WITH SELF-MIXUP TRAINING STRATEGY

  • Exposure bias arises because training denoises with ground-truth context, whereas inference denoises using predicted tokens that may be incorrect.Source: §A.1 · p. 16
  • Self-mixup adds a forward pass in which the model predicts tokens and then denoises based on those predictions.Source: §A.1 · p. 16
  • The authors report that self-mixup enhances sample diversity and attribute this to more accurate inference logits yielding more diverse reasonable sampling paths.They contrast these paths with repeatedly converging on the highest-probability paths.Source: §A.1 · p. 16, Table 8 · p. 16

A.2 DATASET

The appendix describes the training data and optimization setup, then motivates discrete tokens for representing protein structure in language models.

A.2 DATASET

  • DPLM-2's training set combines experimental PDB structures with SwissProt structures filtered to pLDDT > 85, totaling about 200,000 proteins.Training limits length to 512, crops low-confidence SwissProt segments at both ends, and randomly crops long proteins to short lengths with 50% probability per batch.Source: §A.2 · p. 16

A.3 HYPERPARAMETER

  • Models are initialized from pretrained DPLM and trained with AdamW for 100,000 steps, using a maximum learning rate of 1e-4 and diffusion timestep 500.The 150M and 650M models are trained for three days; the 3B model is trained for a week.Source: §A.3 · p. 16

B STRUCTURE TOKENIZER

  • The tokenizer converts protein-backbone 3D coordinates into discrete structure-token sequences, with each token representing a local structural element of a residue.Source: B STRUCTURE TOKENIZER · p. 16

B.1 DATASET

The structure tokenizer and multimodal language model use the same 200K-protein structure dataset. Figure 6 outlines motif-scaffolding evaluation across sequence-based, structure-based, and co-generation settings.

B.1 DATASET

  • The structure tokenizer uses the same experimental and high-quality structure data as the multimodal language model, totaling 200K proteins.Source: §B.1 · p. 16
  • Figure 6 distinguishes sequence-based, structure-based, and co-generation evaluations for motif scaffolding.Source: Figure 6 · p. 17, §B.1 · p. 17
  • The motif-preservation checks include RMSD thresholds below 1.0 and TMScore thresholds above 0.8, depending on the evaluation setting.Source: Figure 6 · p. 17, §B.1 · p. 17

B.2 MODEL ARCHITECTURE

DPLM-2 converts backbone structures into discrete tokens with an encoder–quantizer–decoder tokenizer. Its components combine pretrained geometric features, lookup-free quantization, and an AlphaFold2-derived decoder.

B.2 MODEL ARCHITECTURE

  • A structure encoder, quantizer, and structure decoder form the tokenizer.Source: §B.2 · p. 17
  • The frozen pretrained GVP-Transformer encoder converts backbone structures into geometric features, which an MLP projects into a latent embedding.Source: §B.2 · p. 17
  • A lookup-free quantizer uses a latent dimension of log2 |Z|, where |Z| is the codebook size.Source: §B.2 · p. 17
  • The decoder uses four AlphaFold2 IPA-based EvoFormer layers without MSA row attention to generate atomic positions from structure tokens.Source: §B.2 · p. 17

B.3 TRAINING

Tokenizer training combines structure reconstruction objectives with sequence prediction. The evaluation pipeline assesses overall quality and motif consistency using metrics suited to sequence-based, structure-based, and co-generation approaches.

B.3 TRAINING

  • Tokenizer training combines reconstruction, codebook commitment, and entropy regularization losses within a VQ-VAE framework.Reconstruction uses FAPE, violation, and distogram losses.Source: §B.3 · p. 17
  • A sequence prediction head on the decoder’s final representation is trained against the native sequence using cross-entropy.Source: §B.3 · p. 17

C.1 EVALUATION PIPELINE

  • For sequence-based evaluation, ESMFold predicts structure from the generated sequence and its pLDDT score measures overall quality.Motif-RMSD is calculated from the ESMFold-predicted structure.Source: §C.1 · p. 17
  • Structure-based evaluation predicts a sequence with ProteinMPNN, refolds it with ESMFold, and uses scTM for overall quality; co-generation uses scTM between the generated structure and ESMFold prediction.Source: §C.1 · p. 17

C.2 RESULT OF EACH PROBLEM

DPLM-2 achieves the best average motif-scaffolding success rate across the reported evaluation settings, with comparator wins varying by setting. The surrounding literature describes protein language models and structure-generation methods with distinct modeling approaches.

C.2 RESULT OF EACH PROBLEM

  • DPLM-2 achieves the best average success rate in each evaluation, outperforming ESM3 in 12 co-generation and 10 sequence-based problems, and RFDiffusion in 14 structure-based problems.Source: §C.2 · p. 17, Table 9 · p. 18
  • Selecting the best of eight samples significantly improves results over one sample, especially success rate; the authors hypothesize this alleviates sampling-randomness errors.Source: §C.2 · p. 18

D.1 PROTEIN LANGUAGE MODELS

  • Protein language models can capture evolutionary information about secondary and tertiary structure from sequences alone and generalize across downstream tasks.Source: §D.1 · p. 18

D.2 PROTEIN STRUCTURE GENERATIVE MODELS

  • Structure-generation methods diffuse different representations, including backbone coordinates, inter-residue angles, and residue frames (Trippe et al., 2022; Wu et al., 2022b; Lin & AlQuraishi, 2023; Yim et al., 2023).Source: §D.2 · p. 18
  • Other approaches include distance-and-angle-based ProteinSGM, sequence-generating FoldingDiff, large-protein-and-complex generator Chroma, and multimodal co-generation methods Multiflow and ProtPardelle (Lee et al., 2022; Wu et al., 2022a; Ingraham et al., 2023; Campbell et al., 2024; Chu et al., 2024).Source: §D.2 · p. 19
Loading 2410.13782v1…