PaperlayerView exact revision on arXiv

Source-linked AI summary

BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language

Qizhi Pei, Zhimeng Zhou, Yi Duan, Yiyang Zhao, Wei Li, Han Guo, Liang He, Chengping Li, Chang-Yu Hsieh, Conghui He, Rui Yan, Lijun Wu

arXiv:2606.22138v1cs.CLcs.AIcs.LGq-bio.BM

TL;DR

Existing biological models have not reconciled native multimodality with broad coverage of molecular and protein entities. BioMatrix unifies sequences, structures, and language for molecules and proteins in a decoder-only model, matching or surpassing specialized baselines on 77 of 80 tasks.

  • Problem

    Existing efforts have pursued native multimodality and broad biomolecular entity coverage separately, leaving the two unreconciled within a single architecture.

    Source: §1 · p. 3

  • Method

    BioMatrix uses a single decoder-only architecture to unify sequence, structure, and natural language modalities across molecules and proteins under one next-token prediction objective.

    Source: §1 · pp. 4–5

  • Results

    BioMatrix matches or surpasses specialized baselines on 77 out of 80 tasks across 6 categories.

    Source: §1 · p. 5

  • Takeaways & Limitations

    The results provide empirical evidence that native multimodality and broad entity coverage can be unified without compromising per-task performance.

    Source: §1 · p. 5

    Because molecular and protein structures use disjoint token spaces, BioMatrix cannot natively represent or generate biomolecular complexes, limiting docking and pocket-conditioned ligand-design applications.

    Source: §11 · p. 52

Abstract

from arXiv · show

We present BioMatrix, the first multimodal foundation model that natively integrates sequences, structures, and natural language for both molecules and proteins within a single decoder-only architecture. Existing biological foundation models pursue native multimodality and broad entity coverage separately: those that fuse multiple modalities under a shared objective remain confined to a single entity type, while those spanning multiple entity types either omit explicit structural modeling or rely on adapter-based designs in which the model cannot natively generate the very modalities it can read. BioMatrix closes this gap by mapping molecular sequences (supporting both SMILES and SELFIES notations), molecular structures, protein sequences, protein structures, and natural language into a shared discrete token space through a unified tokenization scheme, so that all modalities are consumed and produced uniformly under a single next-token prediction objective -- without external encoders, projection adapters, or modality-specific output heads. Built upon the Qwen3 language model (1.7B and 4B), BioMatrix is continually pretrained on 304.4 billion tokens spanning general and domain-specific text, sequence and structure views of molecules and proteins, and cross-modal corpora that interleave biomolecular entities with scientific text and link distinct entities through molecule-protein and protein-protein interaction data. After tuning on a comprehensive suite of downstream applications covering 80 tasks across 6 categories -- encompassing single-entity and multi-entity understanding and generation tasks across and within modalities -- BioMatrix achieves state-of-the-art or competitive performance on 77 out of 80 tasks, demonstrating that a single, natively multimodal generalist model can effectively match or surpass specialized approaches across a wide range of biological tasks.

1 Introduction

Biological modeling requires sequences, structures, language, and cross-entity relationships, yet prior work has pursued native multimodality and broad entity coverage separately. BioMatrix brings these capabilities together in one decoder-only model.

Prior work and research gap

  • Prior approaches separate the two goals: adapter-based systems such as BioMedGPT [79] and 3D-MoLM [63] struggle with native structural generation, while BioT5 [95] [96], NatureLM [137], and SciReasoner [132] omit explicit structure.ESM3 models sequence and structure but only for proteins; AlphaFold3, ODesign [149], and LOGOS [61] broaden entity coverage but do not unify language with sequence and explicit structure tokens under one shared interface.Source: §1 · pp. 3–4

Training data

  • Pretraining uses a 304.4-billion-token corpus of scientific text, molecule- and protein-centric multi-view data, interleaved biomedical text, and molecule–protein and protein–protein interactions.Stochastic composition and protein three-view instances expose both isolated and jointly grounded modality views.Source: §1 · pp. 4–5

Empirical results

  • BioMatrix achieves state-of-the-art or competitive performance on 77 of 80 tasks across 6 categories, with its largest gains on tasks exercising the unified design.These include text-based molecule generation, property-conditioned conformer generation, protein sequence–structure co-generation, and structure-grounded molecule–protein affinity prediction.Source: §1 · pp. 4–5

Unified architecture

  • BioMatrix natively integrates molecular and protein sequences, structures, and natural language in one decoder-only architecture, treating single- and multi-entity tasks as conditional generation.Built on Qwen3 (1.7B and 4B), it supports tasks including captioning, text-conditioned design, folding, inverse folding, structure generation, and interaction prediction.Source: §1 · pp. 4–5

Unified tokenization

  • A shared discrete vocabulary covers SMILES, SELFIES, protein sequences, molecular and protein structures, and language, without external encoders or projection adapters.Molecular tokens use a branch-decoupled decoder and frequent SELFIES–structure joint tokens; proteins retain separate residue and GCP-VQVAE structure tokens to preserve alignment without a prohibitively large joint vocabulary.Source: §1 · pp. 4–5

2 Unified Multimodal Tokenization

BioMatrix places natural language, molecular and protein sequences, and molecular and protein structures in a shared discrete token space for a single next-token prediction objective. Dedicated structure tokenizers, an extended Qwen3 vocabulary, and description-based embedding initialization make these modalities usable together in one decoder-only model.

2 Unified Multimodal Tokenization

  • BioMatrix maps language, molecular and protein sequences, and their structures into a shared token space for uniform next-token prediction.Language and sequence-like representations use token sequences; dedicated tokenizers discretize molecular conformations and protein backbones into structural codes.Source: §2 · p. 6

2.1 Molecule Structure Tokenizer

  • Roughly 0.1 Å lower average per-component RMSD on QM9-2014 came from replacing MolStrucTok’s shared decoder with four target-specific MLP heads.The tokenizer retains only generation descriptors; the four heads separately predict length, polar angle, azimuthal magnitude, and azimuthal sign.Source: §2.1 · pp. 6–7

2.2 Protein Structure Tokenizer

  • GCP-VQVAE represents each protein residue with a discrete backbone-geometry code, creating a length-L sequence aligned with amino-acid tokens for folding and inverse folding.Its geometric inputs use backbone atoms and neighboring-frame features; a decoder predicts residue frames to reconstruct backbone coordinates.Source: §2.2 · p. 8

2.3 Vocabulary Extension

  • The extended Qwen3 vocabulary preserves one-token-per-residue protein sequences, composes molecular geometry codes with SELFIES atoms, and factors protein sequence and structure tokens.For molecules, the frequent joint-token prefix covers more than 99% of occurrences with 11,294 entries; protein pairing would require 71,737 entries, so the two token types remain separate. Five start/end token pairs delimit non-text segments, while natural language remains unwrapped.Source: §2.3 · pp. 9–10

2.4 Embedding Initialization

  • Description-based initialization uses Qwen3’s pretrained embeddings to initialize new vocabulary rows and provides a more stable early-pretraining start than random initialization.Descriptions are tokenized with the original Qwen3 tokenizer; each new embedding is their mean pretrained input embedding plus small Gaussian perturbation. Protein residue tokens use amino-acid names, while opaque molecular and structure tokens use their raw strings.Source: §2.4 · pp. 10–11

3 Multimodal Continual Pretraining

BioMatrix is continually pretrained on a 304.4-billion-token mixture spanning text, molecular and protein sequence and structure views, and cross-entity data. Training loss spikes recover without meaningful validation-loss degradation.

3.1 Pretraining Corpus

  • The corpus totals approximately 304.4 billion tokens across general and scientific text, molecule-centric, protein-centric, and cross-entity/interleaved data.Text and protein data contribute the largest shares, followed by molecular and cross-entity data.Source: §3.1 · pp. 11–12, Figure 3 · p. 12

3.1.1 Text Corpora

  • General and scientific text combines FineWeb-Edu, biology-, chemistry-, medicine-, and health-focused FineFineWeb subsets, and full PubMed articles.FineWeb-Edu is used to help prevent catastrophic forgetting of general language abilities; PubMed articles cover molecular biology, pharmacology, and clinical research.Source: §3.1.1 · p. 12

3.1.2 Molecular Data

  • Molecular data combines PubChem representations and annotations, DFT-level conformations, and approximately 2.5M molecule–description pairs from MolTextNet.Instances may combine SMILES, SELFIES, structures, properties, and text under stochastic sampling and permutation of optional fields and component order.Source: §3.1 · p. 11, §3.1.2 · pp. 12–13

3.1.3 Protein Data

  • Protein data spans sequence, structure, and annotated sources, using stochastic composition and three-view instances to expose proteins in isolated and jointly grounded forms.Sources include UniRef50, RCSB PDB, Swiss-Prot with AFDB structures when available, TrEMBL sequences and annotations, and approximately 130M additional AFDB structures covering over 60% of its entries.Source: §3.1 · p. 11, §3.1.3 · p. 13

3.1.4 Cross-Entity and Interleaved Data

  • Cross-entity data pairs biomolecular mentions in scientific text with retrieved representations and adds sequence-level and structure-grounded molecule–protein and protein–protein interactions.Interleaved documents come from PubMed and bioRxiv abstracts, S2ORC, and USPTO-Applications; interaction sources include BindingDB, STITCH, jglaser, CrossDocked2020, AlphaSeq, and PPIRef.Source: §3.1.4 · pp. 13–14

3.2 Training Dynamics

  • Both BioMatrix-1.7B and BioMatrix-4B were pretrained for approximately 36.4K steps, or one epoch, on the full corpus using 64 NVIDIA H100 GPUs.Training used a global batch size of 1,024, maximum sequence length of 8,192 tokens, and AdamW with a peak learning rate of 2.0 × 10−4 under a cosine schedule with 2,000 warmup steps.Source: §3.2 · p. 14
  • Across prominent loss spikes, pre- and post-spike checkpoints had essentially comparable validation loss, and subsequent checkpoints improved smoothly without manual intervention.The spikes occurred for both model sizes, primarily in the early-to-middle phase of training.Source: §3.2 · p. 14

4 Instruction Tuning across Biological Tasks

After multimodal continual pretraining, BioMatrix is instruction-tuned to follow natural-language instructions across heterogeneous biological inputs and outputs. Its corpus spans 80 tasks across molecule, protein, and interaction scopes, with sequence- and structure-based tasks.

4.1 Task Taxonomy and Data Sources

  • Molecular tasks cover sequence-level generation, conversion, prediction, captioning, editing, optimization, and question answering, plus unconditional and property-conditioned conformer generation.Source: §4.1 · p. 15
  • Protein tasks include sequence understanding, annotation, knowledge mining, text-based design, and generation, alongside structure understanding, folding, inverse folding, co-generation, and structure generation.Source: §4.1 · p. 15
  • Interaction tasks cover molecule–protein and protein–protein binding-affinity prediction and interaction modeling in sequence-only and structure-grounded forms.Source: §4.1 · p. 15
  • The instruction-tuning taxonomy comprises 80 public-benchmark tasks across six categories, spanning molecule, protein, and interaction domains and 1D and 3D modalities.MegaScience is included for instruction tuning but excluded from the evaluation suite.Source: §4.1 · p. 15, Table 3 · p. 16
  • MegaScience [27] supplements biology-specific supervision to help preserve broad scientific reasoning and instruction-following inherited from Qwen3.The corpus is included only during instruction tuning, not in the evaluation suite.Source: §4.1 · p. 16, Table 3 · p. 16

4.2 Data Format Unification and Prompt Diversification

  • Data unification serializes biomolecular fields with modality-specific boundary tokens and converts raw examples into instruction-input-response triplets, verbalizing labels and formatting regression targets as numerical strings with task-appropriate units and precision.Source: §4.2 · pp. 16–17
  • To reduce overfitting to fixed instruction wording, each sub-task uses semantically equivalent prompt templates with varied phrasing, sampled uniformly for training instances.Source: §4.2 · p. 17

4.3 Training Configuration

  • Both BioMatrix sizes are tuned with AdamW using LLaMA-Factory, a 5.0 × 10−5 peak learning rate, cosine decay, 10% linear warmup, sample-level attention masking, and a 2,048-token maximum length.Source: §4.3 · p. 17

5 Evaluation Setup

The evaluation tests whether unified tokenization supports structural generation, prediction, and understanding, and whether a single backbone can match or surpass task-specialized models.

5 Evaluation Setup

  • The two evaluation questions are whether structures can share a vocabulary and next-token objective with sequences and language while retaining task quality, and whether the unified model can match or surpass specialists.The questions cover structural generation, prediction, and understanding across molecules and proteins, alongside competitiveness against purpose-built or purpose-trained models.Source: §5 · p. 17
  • Evaluation spans single-entity molecular tasks, single-entity protein tasks, and molecule–protein and protein–protein interaction tasks.The taxonomy follows the instruction-tuning organization and is evaluated across Sections 6–8.Source: §5 · p. 17
  • All reported variants share BioMatrix’s tokenization, continually pretrained backbone, and next-token objective, while instruction tuning uses task-group-specific variants.A single all-tasks merged SFT model is also released, trained on the union of subtask corpora with mild oversampling of small-data subtasks.Source: §5 · pp. 17–18
  • Evaluation follows each benchmark’s official split and metric, using greedy decoding for deterministic-output tasks and top-p = 1.0 nucleus sampling for diversity tasks.Sampling temperature is tuned per task to balance fidelity and diversity; inference uses vLLM [57].Source: §5 · p. 18

6 Molecule Tasks

This section evaluates molecular tasks involving sequence notations and natural language, as well as molecular geometry. BioMatrix performs strongly across sequence-level tasks, with its largest gains on cross-modal tasks and distinct strengths for SMILES and SELFIES.

6 Molecule Tasks

  • BioMatrix achieves the best or second-best result on the vast majority of metrics across sequence-level molecular tasks among LLM-based methods.The suite spans unconditional generation, name conversion, property prediction, captioning, text-based generation, forward and retrosynthesis, editing, customized generation, optimization, and question answering; BioMatrix often surpasses baselines several times larger.Source: §6 · p. 19
  • Roughly 3–4× lower MAE on electronic-structure targets in property-conditioned conformer generation distinguishes BioMatrix from the strongest prior method.Improvements over prior specialists are most pronounced on text-based molecule generation, molecule captioning, and property-conditioned conformer generation, validating the shared discrete token space for sequence, structure, and text.Source: §6 · p. 19
  • SELFIES excels at distribution-level chemical validity tasks, while SMILES excels when tasks reward surface-level structural anchoring.SELFIES dominates unconditional generation and property optimization; SMILES dominates customized generation and synthesis prediction, making support for both notations complementary rather than redundant.Source: §6 · p. 19
  • On unconditional conformer generation, BioMatrix matches diffusion baselines on distributional and dihedral metrics but retains a bond-length precision gap that MMFF refinement mostly closes.This indicates that fine-grained geometric fidelity is limited by quantization error in finite structural codebooks rather than next-token modeling capacity.Source: §6 · p. 19

6.1 1D Molecular Tasks

Across 1D molecular tasks, BioMatrix achieves strong results in generation, prediction, transformation, optimization, and question answering. Performance also depends on representation: SELFIES benefits validity-focused tasks, while SMILES better supports explicit structural constraints.

6.1.1 Unconditional Generation

  • On MOSES, BioMatrix-4B (SELFIES) reaches 0.998 validity, 1.000 uniqueness, 0.925 novelty, IntDiv1 = 0.868, and IntDiv2 = 0.861.On GuacaMol, it reaches 0.998 validity, 1.000 uniqueness, and 0.986 novelty; SELFIES variants guarantee syntactic validity, while Sc2Mol trades validity for novelty and diversity.Source: Table 5 · p. 20, §6.1.1 · pp. 20–21, Table 6 · p. 21

6.1.2 Name Conversion; 6.1.3 Property Prediction

  • 84.40%→92.83% I2S EM over SciReasoner-8B; BioMatrix achieves best or second-best LLM results on five of six property benchmarks and literature SOTA on HIV, SIDER, and 1.7B ESOL.BioMatrix-1.7B surpasses SciReasoner-1.7B on all name-conversion EM metrics and outperforms larger SciReasoner-8B and BioMedGPT-Mol-8B on four property benchmarks; regression results vary more than classification results.Source: §6.1.2 · p. 21, Table 7 · p. 22, §6.1.3 · p. 22

6.1.4 Molecule Captioning and Text-based Molecule Generation

  • 65.07% EM versus SciReasoner-8B’s 48.00%, and 85.47% FTS versus BioMedGPT-Mol-8B’s 77.50%, on text-based molecule generation.BioMatrix leads LLM-based methods on all four metrics and surpasses task-specific literature SOTA on METEOR, EM, and FTS; BioMatrix-1.7B outperforms every LLM baseline on all four metrics.Source: §6.1.4 · pp. 22–23, Table 9 · p. 23

6.1.5 Forward Synthesis and Retrosynthesis

  • 77.94% forward-synthesis EM for BioMatrix-4B (SMILES) exceeds SciReasoner-8B’s 75.00%, while its 45.16% retrosynthesis EM remains competitive with 46.00%.BioMatrix achieves the best or second-best result among LLM-based methods on all six metrics, and SMILES variants consistently outperform SELFIES variants on EM.Source: Table 10 · p. 24, §6.1.5 · p. 24

6.1.6 Molecule Editing

  • 0.9294 SR and 0.6447 WSR on AddComp; BioMatrix-4B (SELFIES) reaches 0.9666 DelComp SR, but BioMatrix trails Claude-3.5 on SubComp.All BioMatrix variants exceed the prior best across both DelComp metrics; substitution is more challenging because it combines deletion and insertion while preserving surrounding context.Source: §6.1.6 · pp. 24–25, Table 11 · p. 25

6.1.7 Customized Molecule Generation

  • 0.2774 AtomNum SR leads; BioMatrix-4B (SMILES) nearly matches the BondNum baseline and is competitive on FuncGroup, while SELFIES falls to 0.0048 FuncGroup SR.SELFIES token semantics obscure atom, bond, and functional-group constraints; its context-dependent tokenization supports syntactic validity in open-ended generation but impedes constraint anchoring.Source: §6.1.7 · pp. 25–27, Table 12 · p. 26

6.1.8 Molecule Optimization

  • BioMatrix leads logP and MR optimization on S2-Bench, while BioMatrix-4B (SELFIES) leads most MolOpt Correct and Validity metric–task combinations, including hERG and strict solubility.BioMatrix remains competitive on S2-Bench QED; SELFIES benefits optimization by generating valid molecules with shifted properties, unlike customized generation, which requires anchoring constraints to surface tokens.Source: §6.1.8 · pp. 27–28, Table 13 · p. 28, Table 14 · p. 29

6.1.9 Molecule Question Answering

  • 73.78% Total accuracy surpasses MolCA-1.3B’s 64.79%, while BioMatrix-4B reaches 83.36% on Structure versus the prior best of 71.12%.BioMatrix-1.7B also outperforms every baseline on Total, Structure, Source, and Application; Property remains hardest, with BioMatrix-4B at 47.88% versus BioT5-base’s 49.11%.Source: Table 15 · p. 30, §6.1.9 · p. 30

6.2 3D Molecular Tasks

In 3D molecular generation, BioMatrix performs strongly on both unconditional molecule-and-conformer generation and property-conditioned conformer generation. Property conditioning produces especially large gains on electronic-structure targets, while raw autoregressive geometry has fidelity limitations that MMFF refinement largely closes.

6.2.1 Unconditional 3D Structure Generation

  • BioMatrix-1.7B attains FCD = 0.064, near the training reference (0.063) and NExT-Mol (0.070), with atom stability, molecule stability, and V&C of 1.000 / 0.988 / 1.000.It is within 0.001–0.002 of the training distribution on Frag (0.991 vs. 0.992) and Scaf (0.944 vs. 0.946), while trailing NExT-Mol on V&U&N (0.749 vs. 0.802) and SNN (0.495 vs. 0.530).Source: Table 16 · p. 31, §6.2.1 · p. 31
  • A single MMFF refinement step improves BioMatrix’s FCD from 1.04 to 0.23 and atom stability from 0.897 to 0.985, closing most of the raw conformer geometry gap.The raw conformers match diffusion-based baselines on dihedral-angle MMD but show bond-length MMD = 1.05; the passage attributes limitations to finite-codebook quantization and accumulated per-atom autoregressive reconstruction errors.Source: §6.2.1 · pp. 31–32

6.2.2 Property-conditioned 3D Structure Generation

  • Conditioned generation evaluates six QM9-2014 quantum properties using predictor MAE against specified targets, with the predictor’s own MAE on a disjoint training half as a lower bound.The auxiliary predictor is trained on Da and BioMatrix on Db; properties are dipole moment, polarizability, heat capacity, HOMO energy, LUMO energy, and HOMO–LUMO gap.Source: §6.2.2 · p. 32
  • BioMatrix-4B achieves the best MAE on five of six conditioned properties and second-best on the remaining target, reducing NExT-Mol’s electronic-target errors roughly 3–4×.For BioMatrix-4B versus NExT-Mol, MAE is 205 →53 meV for εHOMO, 235 →54 meV for εLUMO, and 297 →81 meV for ∆ε; BioMatrix-4B more than halves error on the other three properties. BioMatrix-1.7B surpasses every baseline on all six, and joint tokenization of properties and geometry is identified as an effective conditioning mechanism.Source: Table 17 · p. 32, §6.2.2 · pp. 32–33

7 Protein Tasks

Protein tasks span sequence- and structure-based settings. BioMatrix reports strong performance across protein understanding, folding and inverse folding, generation, and text-based design.

Protein Tasks

  • Across protein understanding, BioMatrix matches or exceeds substantially larger baselines at 1.7B and 4B, while GPT-O3 and DeepSeek-R1 fail to generalize to these domains.The larger baselines include 8B-scale SciReasoner and OPI-Llama variants; tasks include sequence understanding, annotation prediction, and knowledge mining.Source: §7 · p. 34
  • Residue-aligned tokenization makes folding and inverse folding symmetric token-alignment tasks; BioMatrix achieves competitive folding accuracy and state-of-the-art amino acid recovery without specialized architecture or design.Each amino acid token corresponds one-to-one with its per-residue structure token in the shared vocabulary.Source: §7 · p. 34
  • On sequence–structure co-generation and unconditional backbone and sequence generation, BioMatrix matches or surpasses RFDiffusion, FrameDiff, FoldFlow, and DPLM-2 on every standard self-consistency metric.In some cases results approach the native-PDB reference; the advantage holds across decoding orders and conditioning settings.Source: §7 · p. 34
  • For text-based protein design on CAMEO and MolInst-SwissProtCLAP, BioMatrix produces diverse, low-redundancy, structurally plausible sequences while remaining competitive with task-specialized baselines.It does so with a single unified backbone.Source: §7 · p. 34

7.1 1D Protein Tasks

Across protein sequence understanding, annotation, knowledge mining, and sequence generation tasks, BioMatrix performs strongly while supporting both text-conditioned and unconditional protein design.

7.1.1 Sequence Understanding

  • BioMatrix-1.7B reaches 34.34% F1 on CLEAN EC Number Price, beating SciReasoner-8B (22.00%), while BioMatrix-4B achieves 87.25% Family and 30.22% Superfamily Accuracy.On CLEAN EC Number New, BioMatrix-4B has 43.54% F1 versus SciReasoner-8B at 54.00%. BioMatrix-1.7B also leads on Fold-level Accuracy at 16.71%, and BioMatrix-4B reaches 90.69% subcellular-localization Accuracy versus SciReasoner-8B at 91.00%.Source: §7.1.1 · pp. 34–35, Table 18 · p. 35, Table 19 · p. 35

7.1.2 Annotation Prediction

  • BioMatrix-4B records 91.26% Keyword F1 and 86.33% GO term F1, and achieves 79.20% ROUGE-L on General Function; BioMatrix-1.7B scores 73.45% ROUGE-L in Catalytic Activity.BioMatrix-4B also outperforms SciReasoner-8B on challenging stratified OPI test sets and both BioMatrix scales outperform specialized architectures and high-parameter models across protein-related metrics.Source: §7.1.2 · p. 36, Table 20 · p. 37, Table 21 · p. 38

7.1.3 Knowledge Mining

  • BioMatrix-1.7B achieves 41.64% Precision on gSymbol2Tissue, exceeding SciReasoner-8B (41.00%) and OPI-Llama-3.1-8B (40.02%).The Knowledge Mining suite covers tissue prediction and cancer associations from gene symbols or names; generic frontier models show near-zero performance on cancer-related associations, while BioMatrix achieves state-of-the-art cancer-association results.Source: §7.1.3 · pp. 36–38, Table 22 · p. 39

7.1.4 Text-based Protein Design

  • On CAMEO, BioMatrix-1.7B reaches 65.64 mean pLDDT and BioMatrix-4B lowers PPL to 631.89; on MolInstruct-SwissProtCLAP, 4B improves PPL from 581.42 to 482.08.On CAMEO, 1.7B has 50.43% of designs above pLDDT 70, while 4B outperforms 1.7B on ProTrek similarity, keyword recall, and retrieval accuracy. On MolInstruct-SwissProtCLAP, 4B also improves ProTrek similarity (9.24 to 11.70), EvoLlama score (37.56 to 38.40), and retrieval accuracy (23.14 to 31.08); specialist baselines remain ahead on foldability and some alignment metrics.Source: Table 23 · p. 39, §7.1.4 · p. 40, Table 24 · p. 40

7.1.5 Unconditional Sequence Generation.

  • Mean pLDDT reaches 85.28 for BioMatrix-1.7B and 85.01 for BioMatrix-4B, exceeding DPLM-650M (83.25), DPLM-2-650M (82.25), and EvoDiff (35.85).The models generate unconditional amino-acid sequences that are folded with ESMFold for structural-plausibility assessment.Source: §7.1.5 · pp. 40–41

7.2 3D Protein Tasks

Across five 3D protein tasks, BioMatrix uses a shared sequence–structure token interface for understanding, folding, inverse folding, and generation. It delivers strong benchmark results, with particularly high self-consistency in co-generation and unconditional backbone generation and high native-sequence recovery in inverse folding.

7.2.1 Structure Understanding

  • On PFUD, BioMatrix-4B set a state of the art across all four metrics, while BioMatrix-1.7B surpassed ProtTeXLlama3 on BLEU-2 and ROUGE-2.PFUD tests free-form answers to six categories of questions given a protein sequence and backbone structure; the paper attributes performance to continual pretraining on annotated protein corpora, including Swiss-Prot entries paired with predicted structures and functional metadata.Source: Table 25 · p. 41, §7.2.1 · p. 41

7.2.3 Inverse Folding

  • BioMatrix-4B achieved the highest inverse-folding AAR at 75.50%, while its scTM = 0.86 trailed DPLM-2-3B (0.92), ESM3 and MultiFlow (both 0.94).BioMatrix-1.7B reached AAR = 75.20% and scTM = 0.85; both scTM values exceed the 0.5 fold-equivalence threshold, indicating fold agreement despite the native-sequence fidelity advantage.Source: Table 26 · p. 42, §7.2.3 · p. 43

7.2.2 Structure Prediction (Folding)

  • BioMatrix-1.7B (TM = 0.84 / RMSD = 5.49) and BioMatrix-4B (TM = 0.84 / RMSD = 5.38) surpassed smaller DPLM-2 models and closely tracked DPLM-2-3B.Against ESMFold (TM = 0.93 / RMSD = 2.84), a folding specialist, BioMatrix retained a performance gap; its autoregressive outputs were length-matched at inference by sampling and filtering for input length.Source: §7.2.2 · pp. 41–42, Table 26 · p. 42

7.2.4 Sequence–structure Co-generation.

  • In co-generation, BioMatrix-1.7B achieved scTM = 0.979 / scRMSD = 1.50 in struct →seq, exceeding DPLM-2-650M and MultiFlow.In seq →struct, both BioMatrix sizes reached scTM = 0.965, with scRMSD of 2.81 and 2.80, versus 0.907 / 6.34 for DPLM-2-650M; BioMatrix mean pLDDT was 85.01–88.80 versus 82.25–82.69 for DPLM-2-650M.Source: §7.2.4 · pp. 43–44, Table 27 · p. 44

7.2.5 Unconditional Structure Generation

  • In unconditional backbone generation, BioMatrix-4B attained scTM = 0.963 with scRMSD = 3.07, exceeding listed methods on scTM and nearing the native-PDB scTM = 0.969.Its scRMSD trailed RFDiffusion (1.97) but improved on DPLM-2-650M (4.45); the results place BioMatrix on the joint scTM–scRMSD frontier using autoregressive next-token prediction.Source: Table 27 · p. 44, §7.2.5 · pp. 44–45

8 Interaction Tasks

BioMatrix predicts molecule–protein and protein–protein interactions across sequence-only and structure-grounded settings, achieving state-of-the-art or competitive results by processing heterogeneous entities in one token stream without pair-specific architecture.

8.1.1 Molecule–protein interaction.

  • RMSE is 1.030 for BioMatrix-4B on BindingDB and 1.268 for both BioMatrix variants on PDBBind/CASF-2016 sequence-based affinity prediction.The BindingDB result beats literature SOTA (1.340), while PDBBind/CASF-2016 remains competitive with literature SOTA (1.181); no task-specific architecture is used.Source: Table 28 · p. 46, §8.1.1 · p. 46

8.1.2 Protein–protein Interaction.

  • BioMatrix-4B scores 62.18% ACC on Yeast, 83.12% ACC on Human, and RMSE 2.350 on PPI Affinity.It exceeds non-pretrained baselines on all three sequence-only tasks and remains competitive with ESM-1b on classification and ProtBert on affinity.Source: Table 28 · p. 46, §8.1.2 · pp. 46–47

8.2.1 Molecule–protein Interaction.

  • RMSE 1.260, Pearson 0.737, and MAE 0.972 make BioMatrix-4B best on three PDBBindv2020 metrics; BioMatrix-1.7B leads on Spearman at 0.717.BioMatrix-4B is second on Spearman (0.712); both variants use shared structure tokens without explicit pocket encoding, geometric cross-attention, or a fusion module.Source: Table 29 · p. 47, §8.2.1 · p. 47

9 Discussion

Across 80 tasks, BioMatrix’s unified tokenization is most valuable for cross-modal and cross-entity work, while task-group tuning and notation choice matter, scaling gains vary, and fine-grained geometry remains limited by tokenization.

9.1 Why Train Task-Group Variants Rather Than One Monolithic SFT Model

  • Mixing heterogeneous biological tasks in one SFT run rarely yields mutual gains and can destabilize small-data tasks, so task-group variants are used.Biological instances tend to share schemas within subtasks, allowing gradients from large-data subtasks to overwhelm small ones; the released all-tasks model uses mild oversampling but still trails task-group variants on a non-trivial fraction of small-data tasks.Source: §9.1 · p. 48

9.2 SMILES vs. SELFIES: Complementary, Not Interchangeable

  • SELFIES favors validity-oriented generation, SMILES favors surface-anchored tasks, and the notations are near-neutral on several other molecular tasks.On unconditional generation, SELFIES variants attain 0.995–0.999 validity on MOSES and GuacaMol versus 0.951–0.994 for SMILES; on MolOpt-Instructions, BioMatrix-4B (SELFIES) reaches 0.98–1.00 validity across all eight subtasks, versus 0.71–0.97 for SMILES, and dominates Correct under loose and strict criteria. Conversely, customized-generation FuncGroup SR is 0.2714 for BioMatrix-4B (SMILES) versus 0.0048 for SELFIES, while forward-synthesis/retrosynthesis EM is 77.94/45.16 versus 69.08/42.90; SELFIES grammar supports validity, whereas its context-dependent tokens obscure surface patterns. Property prediction, captioning, text-based generation, molecule editing, single-property S2-Bench optimization, and MoleculeQA are near-neutral.Source: §9.2 · pp. 48–49

9.3 Where Scaling from 1.7B to 4B Helps, and Where It Does Not

  • 79.45 →94.09 I2F EM for SMILES and 56.35 →65.07 EM for SMILES illustrate strong scaling gains on knowledge recall and structured generation.Forward synthesis EM rises from 72.75 to 77.94; other gains include optimization, MoleculeQA, and protein knowledge mining. Unconditional generation, protein annotation, structure-QA, sequence understanding, and inverse-folding AAR are largely saturated or near parity, while ESOL RMSE worsens from 0.810 to 1.240 and some small-data tasks show non-monotone, high-variance results.Source: §9.3 · pp. 49–51

9.4 Native Multimodality Pays Off Most on Cross-Modal and Cross-Entity Tasks

  • BioMatrix’s unified token space yields its clearest advantages on cross-modal and cross-entity tasks, while matching rather than dramatically exceeding specialists on many single-modality tasks.The gains widen on property-conditioned conformer generation, text-based molecule generation, and sequence- or structure-based molecule–protein affinity prediction; the authors attribute this to making these tasks tractable under one next-token objective without bolt-on fusion modules.Source: §9.4 · p. 51

9.5 Tokenization Is the Bottleneck on Fine-Grained 3D Geometry

  • A single MMFF refinement improves QM9 FCD from 1.04 to 0.23 and atom stability from 0.897 to 0.985, closing most of the fine-grained geometry gap.BioMatrix matches or surpasses diffusion specialists on distribution-level or backbone-topology tasks, but bond-length MMD and folding RMSD still trail specialists; the authors trace these gaps to finite-codebook quantization and autoregressive coordinate reconstruction. They propose geometric post-processing as a practical complement and larger or hierarchical codebooks, side-chain tokenization, or global non-autoregressive decoding as possible improvements.Source: §9.5 · pp. 51–52

10 Conclusion

BioMatrix unifies molecular and protein sequences, structures, and natural language in one decoder-only model, trained to handle modalities uniformly and achieving state-of-the-art or competitive performance on most evaluated benchmarks.

10 Conclusion

  • BioMatrix achieves state-of-the-art or competitive performance on the vast majority of evaluated benchmarks, matching or surpassing specialized models across biological tasks.The conclusion attributes this performance to continual pretraining on a large-scale multimodal corpus and instruction tuning across a comprehensive task suite.Source: §10 · p. 52
  • BioMatrix natively integrates sequences, structures, and natural language for molecules and proteins in a single decoder-only architecture.All modalities share a discrete token space and are handled under one next-token prediction objective, without external encoders, projection adapters, or modality-specific output heads.Source: §10 · p. 52
  • BioMatrix is presented as a step toward unified biological foundation models that bridge sequences, structures, and language across diverse biomolecular entities.Source: §10 · p. 52

11 Limitations

BioMatrix’s limitations concern cross-entity 3D modeling, possible pretraining–evaluation overlap, general-purpose capability retention, SFT balance, and coverage of biological entities.

Cross-entity 3D structures

  • Separate molecular and protein 3D tokenizers prevent BioMatrix from representing or generating complexes, limiting docking and pocket-conditioned ligand design.The paper identifies a shared-frame unified 3D tokenizer as the next step.Source: §11 · p. 52

Pretraining and evaluation overlap

  • No dedicated entity-level filtering means SFT or evaluation entities and annotations may have been implicitly observed during continual pretraining.The authors frame this as a practical limitation of public biological corpora and do not claim a strictly contamination-free benchmark setting.Source: §11 · p. 52

General-purpose capabilities

  • Despite including FineWeb-Edu and MegaScience, biomolecular specialization shifts BioMatrix from Qwen3-Base’s general capabilities; those capabilities were not systematically benchmarked.The authors expect a measurable gap on general-purpose instruction-following or reasoning tasks.Source: §11 · p. 52

SFT model design

  • Task-group SFT variants outperform the unified all-tasks model on a non-trivial fraction of small-data subtasks, and the authors have not fully solved this imbalance.Naively mixing pattern-monolithic biological SFT data lets large-data subtasks overwhelm small ones; balancing, curriculum design, or tailored training dynamics remain open directions.Source: §11 · p. 53

Entity and modality coverage

  • BioMatrix currently covers small molecules and proteins, but not nucleic acids, carbohydrates, or lipids.The authors describe extending the unified tokenization scheme to these entities and modalities as a natural direction.Source: §11 · p. 53

Appendix

The appendix documents BioMatrix’s instruction-tuning grouping and reports training and test statistics across molecule, protein, generation, interaction, and text subtasks.

Appendix

  • Table 30 is identified as the instruction-tuning grouping of BioMatrix.The supplied passage gives the table’s scope but not its grouping contents.Source: Table 30 · p. 62
  • Table 31 reports per-subtask training and test statistics, including 220,285 training and 500 test examples for PDB/AFDB unconditional sequence generation [131].The appendix also identifies Table 30 as the instruction-tuning grouping of BioMatrix.Source: Table 31 · p. 63, Appendix · p. 65
  • MolOpt-Instructions lists 154,492 training and 8,583 test examples for acceptor+ [144], while S2-Bench lists 133,333 training and 5,000 test examples for logP [60].Source: Appendix · p. 64
  • MoleculeQA includes 32,176 training and 3,113 test examples for Structure and 11,062 training and 1,343 test examples for Source [78].Its listed subtasks also include Property and Usage.Source: Appendix · p. 64
  • Mol-Instructions lists 110,689 training and 3,494 test examples for Protein Function and 83,939 training and 2,633 test examples for General Function [28].The listed protein subtasks also include Catalytic Activity and Domain Motif.Source: Appendix · p. 65
  • The appendix lists PFUD structure understanding with 391,771 training and 7,702 test examples, and PDB/AFDB folding with 220,285 training and 447 test examples [82] [131].PDB/AFDB tasks also include inverse folding (220,285 training, 447 test), sequence–structure co-generation (220,285 training, 500 test), and unconditional structure generation (220,285 training, 500 test) [131].Source: Appendix · p. 65
  • Interaction statistics include 7,900 training and 5,230 test BindingDB molecule–protein examples and 35,669 training and 237 test Human protein–protein examples [140].The appendix also lists PDBBindv2019, Yeast, PPI, and 3D PDBBindv2020 interaction tasks.Source: Appendix · p. 65
  • The listed text corpus is MegaScience, with 1,253,230 training examples and no test count reported [27].The table uses dashes for the test count.Source: Appendix · p. 65

Related papers

Loading 2606.22138v1…