PaperlayerView exact revision on arXiv

Source-linked AI summary

Trion-1: Unifying Language, Protein Sequence, and Structure with Diffusion Language Models

Tianyu Lu, Kaicheng Yang, Jiangtao Feng, Keyue Qiu, Yawen Ouyang, Yixin Wu, Xinbo Zhang, Lihao Wang, Wei-Ying Ma, Hao Zhou

arXiv:2610.31215v1cs.AI

TL;DR

Protein modeling remains fragmented across specialized systems, while a general-purpose protein model must connect sequence, structure, and language and retain native LLM interaction. Trion-1 unifies these modalities in a diffusion language model spanning six capabilities. It achieves broadly competitive benchmark performance, including strong unified folding results at a single-sample inference budget.

  • Problem

    Protein modeling remains fragmented across specialized systems and largely disconnected from general-purpose LLMs, despite the need to share knowledge across tasks and retain native language capabilities.

    Source: §1 · p. 2

  • Method

    Trion-1 combines a shared discrete token space for sequence, structure, and language with a non-causal diffusion backbone in one model spanning six protein capabilities.

    Source: §1 · p. 2

  • Results

    At 4B scale, Trion-1 reaches TMbb 0.84 with one unified MDLM checkpoint, comparable to prior tri-modal models using a 200× larger inference budget.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Trion-1 demonstrates that broad understanding, generation, and native language interaction can coexist with strong structure modeling in one model.

    Source: §6 · p. 11

    Trion-1 currently models proteins only, not complexes, ligands, or nucleic acids, and remains behind the strongest task-specialized models, particularly in folding.

    Source: §5 · p. 11

Abstract

from arXiv · show

We envision a general-purpose protein foundation model as a unified, interactive model of sequence, structure, and function, capable of understanding, generation, and cross-modal translation while retaining the instruction-following and reasoning capabilities of modern LLMs. Toward this vision, we introduce Trion-1, the first native, unified, multimodal diffusion language model (dLLM) for proteins. Trion-1 is built on two key design choices: (1) a unified discrete token space that jointly represents sequence, structure, and natural language; (2) a dLLM backbone that replaces strict left-to-right factorization with bidirectional or semi-bidirectional token interactions, enabling global residue communication and long-range contact modeling that are naturally suited to structure modeling. This formulation unifies six core capabilities: folding, inverse folding, sequence understanding, structure understanding, sequence design, and structure design, while retaining the native language capabilities of general-purpose LLMs. On established benchmarks, Trion-1 achieves broadly competitive performance. In folding, Trion-1 achieves a TM-score above 0.8 with a single sample under strict homology control, matching or exceeding the existing tri-modal model using 200 samples. For sequence understanding, Trion-1 performs on par with top-tier LLMs such as Claude Opus 4.7 (62.37%). These results support well-aligned sequence-structure-language representations and strong cross-modal generative capabilities.

1 Introduction

Trion-1 addresses the fragmentation of protein modeling across specialized systems and its disconnection from general-purpose LLMs. It unifies sequence, structure, and language in a diffusion model spanning six protein capabilities.

1 Introduction

  • Protein modeling remains fragmented across specialized systems and largely disconnected from general-purpose LLMs.The paper identifies a need for knowledge sharing across tasks and retention of instruction-following, reasoning, and interactive language capabilities.Source: §1 · p. 2
  • Earlier tri-modal autoregressive models remained limited on structure-intensive tasks, leaving open whether left-to-right generation is suitable for unified protein modeling.The paper argues that structural contacts lack a left-to-right order, so early predictions may have incomplete context and accumulate errors.Source: §1 · p. 2
  • Trion-1 combines a shared discrete token space for sequence, structure, and language with a non-causal diffusion backbone for global residue interactions.The framework unifies folding, inverse folding, sequence understanding, structure understanding, sequence design, and structure design, with native language interaction and interleaved composition.Source: §1 · p. 2
  • At 4B scale, Trion-1 reaches TMbb 0.84 with one unified MDLM checkpoint, comparable to prior tri-modal models using a 200× larger inference budget.On GeneralQA, it performs on par with Claude Opus 4.7 and AMix2 [10].Source: §1 · p. 2

2 Related Work

Related work covers diffusion language modeling, specialized protein systems, and unified tri-modal models. It frames Trion-1 against sequence-ordered generation and prior approaches to combining modalities.

2.1 Diffusion Language Models

  • Diffusion language models generate through iterative denoising, allowing token prediction with bidirectional context rather than fixed left-to-right factorization.Recent advances have enabled dLLMs to rapidly close the performance gap with autoregressive LLMs [12].Source: §2.1 · p. 3

2.2 Specialized Model

  • Specialized protein systems use geometric or language-aware inductive biases across folding, inverse folding, understanding, and design.Folding systems use triangular residue interactions for geometric consistency and long-range contacts; inverse-folding models encode backbones into geometry-aware representations.Source: §2.2 · p. 3

2.3 Unified Tri-modal Models

  • Prior unified models range from contrastive alignment to generative systems, including ESM3 and the autoregressive tri-modal BioMatrix [49].BioMatrix uses a single AR LLM backbone, while this work investigates the analogous unified formulation with a dLLM.Source: §2.3 · p. 3

3 Methods

The methods place sequence, structure, and language in a shared token space and model them with one diffusion backbone. Training proceeds from sequence-language adaptation to multimodal supervision across six cross-modal directions.

3.1 Shared Embedding

  • Trion-1 adopts the DPLM2 structure tokenizer, with codebook-free quantization presented as convenient rather than required.The broader criterion is that tokenizer geometry be faithfully represented in the language-model embedding space.Source: §3.1 · pp. 4–5

3.2 Unified Architecture

  • A single dLLM backbone models sequence, backbone structure, and natural language, which can each serve as conditions or generation targets.The MDLM variant uses fully non-causal masked diffusion; the BDLM variant is bidirectional within blocks and causal across blocks.Source: §3.2 · p. 5

3.3 Multi-stage Training

  • Training starts from Qwen3-4B, acquires protein knowledge through AR-CPT, adapts to diffusion through A2D-CPT, then uses SFT for all six directed cross-modal transformations.Sequence is aligned with language during continual pre-training; structure is introduced during SFT because structural data are scarce.Source: §3.3 · p. 5, Table 1 · p. 5

4 Experiments

Experiments evaluate Trion-1 on folding, understanding, and text-conditioned design, then probe its token geometry and folding attention. Results show a folding advantage over unified autoregressive models alongside remaining design and geometric-consistency challenges.

4.1.1 Evaluation Settings

  • Evaluations cover folding and inverse folding, ProteinArena and OPI-struct analysis tasks, and ProteinArena design, with specialized protein models and frontier LLMs among the baselines.AlphaFold-family configurations requiring MSA inputs are excluded for input comparability.Source: §4.1.1 · p. 6

4.1.2 Folding and Inverse folding

  • TMaa rises from 0.60 to 0.86 and RMSDaa falls from 12.62 to 5.64 versus BioMatrix@1; Trion-1 (MDLM) reaches TMaa 0.86 versus BioMatrix’s metric-wise best-of-200 0.82.The comparison evaluates folding and inverse folding using CAMEO 2022 and PDB-date benchmarks.Source: Table 2 · p. 6, §4.1.2 · pp. 6–7

4.1.3 Protein Understanding

  • 62.16% GeneralQA accuracy closely matches Claude Opus 4.7 at 62.37%, while backbone-only StructureMCQA reaches 76.72%, below STELLA’s 80.56%.STELLA and BioMatrix’s published structure-understanding evaluations use sequence plus structure inputs, unlike Trion-1’s backbone-only evaluation.Source: Table 3 · p. 7, §4.1.3 · p. 7

4.1.4 Protein Design

  • 70.65 vs. 60.32 pLDDT and 47.50 vs. 31.40 IPRRecovery improve over BioMatrix, but Trion-1 remains below AMix2-dLLM across SR, pLDDT, and IPRRecovery.Quality comparisons are conditional on each model’s evaluated subset.Source: Table 4 · p. 8, §4.1.4 · p. 8
  • Within these models, BDLM consistently improves over MDLM in foldability, functional recovery, and repetition control; without adaptive padding, MDLM tends to fill the inference budget, potentially exacerbating repetitive continuations and reducing sequence quality.Source: §4.1.4 · p. 8
  • 0.875–0.921 scTMbb scores accompany reference-like PrcB folds with local variations, but structural self-consistency alone does not establish functional fidelity.Figure 3 enlarges the reference; the four designs share its scale and colors distinguish helices, sheets, loops, and the aligned AlphaFold reference.Source: §4.1.4 · p. 9, Figure 3 · p. 9

4.2.1 Embedding Geometry

  • Figure 4 reports monotonically declining embedding similarity with Hamming distance and an organized structure-token manifold sharing embedding space with sequence tokens.The reported geometry also includes ring-like manifolds and amino-acid tokens near characteristic structural neighborhoods.Source: §4.2.1 · p. 9, Figure 4 · p. 10

4.2.2 Probing the Folding Mechanism

  • Similar diagonal attention persists under global positional encoding, suggesting folding-task residue-level translation; causal masks block direct access to future partner states.The proposed pathway is sequence → contact constraints → structure.Source: Figure 5 · p. 10, §4.2.2 · p. 10

5 Limitation and Future Direction

Trion-1 is currently limited to protein-only modeling and remains behind the strongest task-specialized models on structure-intensive tasks. The authors identify design refinements and test-time scaling as future directions.

5 Limitation and Future Direction

  • Trion-1 does not yet cover complexes, ligands, or nucleic acids, and remains behind the strongest task-specialized models, particularly in protein folding.The current scope is protein-only; the performance gap is noted despite strong performance across diverse tasks.Source: §5 · p. 11
  • Adaptive length and format control, repetition reduction, and functional fidelity remain areas for further design-specific refinement.Quality comparisons are conditional on each model’s evaluated subset.Source: §5 · p. 11
  • The authors propose multi-codebook structure representations, stronger structure decoders, and targeted augmentation for short proteins and challenging cases to raise the capability ceiling.They also propose iteratively generating, evaluating, and refining composable capabilities at test time; sufficiently strong base capabilities could support UniProt-scale corpora for closed-loop self-improvement.Source: §5 · p. 11

6 Conclusion

Trion-1 unifies protein sequence, structure, and natural language in a generative framework that treats six core capabilities as conditional generation. The conclusion reports broadly competitive 4B-scale performance, stronger folding among unified protein models, and composable workflows.

6 Conclusion

  • Trion-1 brings protein sequence, structure, and natural language into a single generative framework.It casts six core capabilities across the sequence–structure–function space as conditional generation.Source: §6 · p. 11
  • At 4B scale, Trion-1 achieves broadly competitive performance while substantially strengthening folding in unified protein models.The conclusion says strong structure modeling coexists with broad understanding, generation, and native language interaction in one model.Source: §6 · p. 11
  • The analyses support non-causal modeling for protein structure and show that structure-tokenizer geometry can be absorbed into the language-model embedding space.Source: §6 · p. 11
  • Trion-1 turns specialized protein capabilities into composable generative primitives for interleaved workflows and future test-time scaling and closed-loop self-improvement.The proposed future approaches use additional inference-time computation and self-generated data.Source: §6 · p. 11

A Diffusion Preliminaries

The appendix introduces notation and describes masked diffusion’s forward corruption, reverse denoising, training objective, block-wise variant, and conditional use across modalities.

A Diffusion Preliminaries

A.1 Forward Transition Process

  • The forward process independently retains each token with probability α_t or replaces it with an absorbing mask state with probability 1 − α_t.With the linear schedule α_t = 1 − t, each position is masked with probability t, and the sequence is fully masked at t = 1.Source: §A.1 · p. 17

A.2 Reverse Denoising Process

  • The reverse ancestral update preserves visible tokens and progressively reveals masked tokens as time decreases from 1 to 0.For masked positions, the update uses the predicted clean-token probability vector described in the appendix.Source: §A.2 · p. 18

A.3 Masked Diffusion Language Modeling Objective

  • Only masked positions contribute to the reconstruction loss, and the predictor has bidirectional access to visible tokens.Time is implicit in the corruption; explicit time conditioning is not required by the notation.Source: §A.3 · p. 18

A.4 Block-Wise Diffusion

  • Block diffusion combines autoregressive ordering across blocks with bidirectional token interactions within each current block.Preceding blocks are fixed context, while future blocks remain inaccessible; training uses clean preceding blocks and inference uses completed generated blocks.Source: §A.4 · p. 18

A.5 Conditional Generation Across Modalities

  • Conditional generation uses the supplied modality and task instruction as context and the target modality as the sequence to generate.For folding, sequence is context and backbone tokens are target; inverse folding exchanges those roles.Source: §A.5 · p. 18

B SFT Dataset Details

The appendix presents the task templates and corpus used for supervised fine-tuning.

B SFT Dataset Details

B.1 Task Templates

Table 6 presents SFT examples for six directed tasks across sequences, backbones, and text. Examples use a shared user–assistant format, with reasoning and final-answer spans indicated where applicable.

B.1 Task Templates

  • Table 6 shows examples of six directed tasks over sequences (S), backbones (B), and text (T).Modality and motif special tokens are omitted for clarity; italicized placeholders replace biological token spans, and {motif } marks a segment within a generated sequence.Source: §B.1 · p. 19, Table 6 · p. 19
  • Examples use a shared user–assistant format, with role delimiters omitted.CoT marks whether the source instance includes reasoning, and Answer shows the final response after that reasoning span.Source: §B.1 · p. 19

B.2 Dataset Statistics

Figure 6 summarizes the 18.5B-token SFT corpus by task and source before further dataset sampling during training.

B.2 Dataset Statistics

  • 18.5B tokens make up the SFT corpus summarized in Figure 6 before further dataset sampling during training.Corpus proportions therefore differ from the effective training mixture.Source: §B.2 · p. 19
  • Figure 6 labels show actual token counts and shares, but small slices are enlarged to at least 5° for readability, so slice areas are not proportional to token counts.B denotes billion and M denotes million.Source: Figure 6 · p. 20

C Benchmark Evaluation Protocol

The benchmark protocols define task-specific datasets, metrics, aggregation, and scoring rules for folding, inverse folding, sequence understanding, structure understanding, and sequence design.

Folding

  • Folding evaluation covers 632 monomeric targets and reports per-protein mean TMca, TMbb, and RMSDbb within each split.TMca uses Cα coordinates; TMbb uses backbone N/Cα/C coordinates; RMSDbb is computed after a single Kabsch alignment, and higher TM scores but lower RMSDbb indicate better predictions.Source: C Benchmark Evaluation Protocol · p. 20

Inverse folding

  • Inverse folding uses the same 183 CAMEO 2022 and 449 PDB targets, scoring amino-acid recovery and self-consistency TMbb.Each generated sequence is folded once with ESMFold v1; unavailable folding scores are excluded from scTMbb aggregation while AAR is retained and metric coverage recorded.Source: C Benchmark Evaluation Protocol · p. 20

Sequence understanding

  • GeneralQA evaluates 481 questions with binary biological-equivalence judgments, and overall accuracy pools all questions rather than averaging category percentages.The questions span Function & Family, Interaction & Binding, Location & Modification, and Other.Source: C Benchmark Evaluation Protocol · p. 20

Structure understanding

  • StructureMCQA scores exact-choice accuracy on 3,918 four-option OPI-Struc questions, while BioMatrix uses a separately marked nearest-option protocol.BioMatrix receives sequence and structure inputs, whereas Trion-1 receives backbone inputs.Source: C Benchmark Evaluation Protocol · pp. 20–21

Sequence design

  • Sequence design evaluates 870 InterPro-based prompts, with success rate measuring accepted sequences for evaluation rather than functional correctness.Quality metrics use finite per-sequence scores; repetition is lower-is-better, while pLDDT, functional recovery, and novelty are higher-is-better. The table uses the first stored candidate per prompt, except Pinal’s marked likelihood-ranked best-of-ten; failed parses remain in the success-rate denominator.Source: C Benchmark Evaluation Protocol · p. 21

Related papers

Loading 2610.31215v1…