PaperlayerView exact revision on arXiv

Source-linked AI summary

BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language

Qizhi Pei, Zhimeng Zhou, Yi Duan, Yiyang Zhao, Wei Li, Han Guo, Liang He, Chengping Li, Chang-Yu Hsieh, Conghui He, Rui Yan, Lijun Wu

arXiv:2606.22138v1cs.CLcs.AIcs.LGq-bio.BM

TL;DR

Existing biological foundation models have not jointly reconciled native multimodality with broad coverage of molecules and proteins. BioMatrix unifies sequences, structures, and language for both entity types in one decoder-only model, achieving state-of-the-art or competitive performance on 77 of 80 tasks. Its results support a single generalist backbone across diverse biological applications.

  • Problem

    Existing models have pursued native multimodality and broad biomolecular entity coverage separately rather than reconciling both within one architecture.

  • Method

    BioMatrix maps molecular and protein sequences, structures, and natural language into one discrete token space for unified next-token prediction in a decoder-only architecture.

  • Results

    77 out of 80 tasks achieved state-of-the-art or competitive performance across 80 tasks spanning 6 categories.

  • Takeaways & Limitations

    A single natively multimodal generalist backbone can match or surpass specialized models across a wide range of biological tasks.

  • Takeaways & Limitations

    Because molecular and protein structures use disjoint tokenizers and geometric reference frames, BioMatrix cannot natively represent or generate biomolecular complexes.

Abstract

from arXiv · show

We present BioMatrix, the first multimodal foundation model that natively integrates sequences, structures, and natural language for both molecules and proteins within a single decoder-only architecture. Existing biological foundation models pursue native multimodality and broad entity coverage separately: those that fuse multiple modalities under a shared objective remain confined to a single entity type, while those spanning multiple entity types either omit explicit structural modeling or rely on adapter-based designs in which the model cannot natively generate the very modalities it can read. BioMatrix closes this gap by mapping molecular sequences (supporting both SMILES and SELFIES notations), molecular structures, protein sequences, protein structures, and natural language into a shared discrete token space through a unified tokenization scheme, so that all modalities are consumed and produced uniformly under a single next-token prediction objective -- without external encoders, projection adapters, or modality-specific output heads. Built upon the Qwen3 language model (1.7B and 4B), BioMatrix is continually pretrained on 304.4 billion tokens spanning general and domain-specific text, sequence and structure views of molecules and proteins, and cross-modal corpora that interleave biomolecular entities with scientific text and link distinct entities through molecule-protein and protein-protein interaction data. After tuning on a comprehensive suite of downstream applications covering 80 tasks across 6 categories -- encompassing single-entity and multi-entity understanding and generation tasks across and within modalities -- BioMatrix achieves state-of-the-art or competitive performance on 77 out of 80 tasks, demonstrating that a single, natively multimodal generalist model can effectively match or surpass specialized approaches across a wide range of biological tasks.

1 Introduction

BioMatrix addresses the unresolved integration of native multimodality and broad biomolecular entity coverage in one decoder-only model. It unifies sequences, structures, language, and cross-entity relationships through shared discrete tokens and demonstrates strong performance across 80 tasks.

  • Motivation and gap: Existing biological foundation models have pursued multimodality and broad entity coverage separately, leaving their integration within one architecture unresolved.Multimodal models remain confined to a single entity type, whereas broader models may lack natural language or explicit structural modeling.
  • BioMatrix: BioMatrix natively integrates molecular and protein sequences, structures, natural language, and interactions in a single decoder-only architecture under one next-token objective.The model supports both single- and multi-entity understanding and generation.
  • Unified tokenization: A unified tokenization scheme maps SMILES, SELFIES, protein sequences, molecular and protein structures, and natural language into one shared discrete vocabulary without external encoders or projection adapters.This enables heterogeneous biomolecular inputs and outputs to be handled uniformly.
  • Training and evaluation: 304.4 billion tokens span within-modality and cross-modality data, while instruction tuning covers 80 tasks across 6 categories.The pretraining corpus includes general and domain-specific text, biomolecular sequence and structure views, and cross-modal entity relationships.
  • Results: 77 out of 80 tasks achieve state-of-the-art or competitive performance, showing that the unified backbone can match or surpass specialized baselines across diverse biological applications.The evaluated tasks include molecular, protein, structural, language, and biomolecular interaction understanding and generation.

2 Unified Multimodal Tokenization

BioMatrix uses one unified tokenization scheme to map molecular and protein sequences, structures, and natural language into a shared discrete space for a single next-token prediction objective. Dedicated structural tokenizers encode molecular conformations and protein backbones, while Qwen3 tokenization handles language and sequence representations.

  • Unified design: All modalities are mapped into a shared discrete token space and processed by the same decoder-only next-token prediction objective.This design covers sequences, structures, and natural language for both molecules and proteins.
  • Molecular structures: Molecular structures are encoded as per-atom structural tokens from a 512-entry codebook aligned one-to-one with SELFIES atom tokens.MolStrucTok discretizes geometric descriptors using a VQ-VAE, and BioMatrix integrates the resulting codes with corresponding SELFIES tokens.
  • Molecular structures: A branch-decoupled molecular decoder reconstructs heterogeneous geometric descriptor components and improves reconstruction fidelity.The decoder replaces MolStrucTok’s original shared-trunk, single-head design.
  • Protein structures: Protein backbones are represented as length-L sequences of one-per-residue structural tokens from a 4,096-entry codebook, aligned directly with amino-acid sequences.BioMatrix adopts GCP-VQVAE Large as-is; its pose-invariant codes support folding and inverse-folding generation.
  • Sequence and language tokenization: Molecular language and sequences use Qwen3 tokenization, with added vocabulary entries for SELFIES while retaining SMILES and SELFIES as parallel representations.Natural language reuses the Qwen3 tokenizer, SMILES is tokenized directly, and SELFIES bracketed atomic symbols are added as new entries.
  • Vocabulary initialization: Description-based initialization reuses the frozen Qwen3 embedding table and provides a more stable early continual-pretraining starting point than random initialization.No additional training or auxiliary models are required for this initialization scheme.

3 Multimodal Continual Pretraining

BioMatrix’s continual pretraining uses a heterogeneous 304.4-billion-token corpus spanning four data categories and sequence, structure, and language modalities for molecules and proteins. Unified tokenization, stochastic composition, and three-view instances expose entities to modalities independently, jointly, and through cross-entity links.

  • Corpus composition: The corpus contains general and scientific text, molecule-centric data, protein-centric data, and cross-entity or interleaved data.These categories cover within-entity multimodal views and links between distinct biomolecular entities or scientific prose.
  • Instance construction: Molecule- and protein-centric instances use stochastic composition to sample and order fields, while a three-view pattern emits multiple modality combinations per entity.The strategies support handling aligned modalities independently and jointly.
  • Corpus composition: 304.4 billion tokens span general and domain-specific text, molecular and protein sequence and structure views, and cross-entity resources.Text and protein data contribute the largest shares, followed by molecular and cross-entity data.
  • Molecule-centric data: Molecular data combine canonical SMILES, SELFIES, tokenized conformations, properties, and descriptions, supplemented by DFT-level conformations and approximately 2.5M molecule–description pairs.Sources include PubChem, PCQM4Mv2, PubChemQC, and MolTextNet.
  • Protein-centric data: Protein data cover broad sequences, experimentally determined and predicted structures, and functional annotations from UniRef50, RCSB PDB, UniProt, and AFDB.Swiss-Prot provides richly annotated proteins, while TrEMBL broadens taxonomic and functional sequence coverage.
  • Cross-entity and interleaved data: Cross-entity data interleave biomolecular representations with biomedical text and include sequence- and structure-grounded molecule–protein and protein–protein interaction pairs.The interleaved corpus draws from PubMed, bioRxiv, S2ORC, and USPTO-Applications, while interaction data include BindingDB, STITCH, CrossDocked2020, and other resources.

4 Instruction Tuning across Biological Tasks

BioMatrix is instruction-tuned across 80 heterogeneous biological tasks spanning molecules, proteins, interactions, sequences, and structures. The pipeline unifies datasets through modality serialization, instruction conversion, diversified prompts, and specified training configurations.

  • 4.1 Task Taxonomy and Data Sources: The instruction-tuning corpus comprises 80 tasks spanning molecule, protein, and interaction entities across sequence-based and structure-based modalities.The taxonomy is organized into six task categories and draws from public benchmarks.
  • 4.1 Task Taxonomy and Data Sources: Molecular coverage includes sequence generation, conversion, prediction, captioning, design, optimization, question answering, and structure-based conformer generation.The sequence tasks include forward and retrosynthesis, editing, customized generation, and property-conditioned conformer generation.
  • 4.1 Task Taxonomy and Data Sources: Protein coverage includes sequence understanding, annotation, knowledge mining, design, generation, structure understanding, folding, inverse folding, and sequence–structure co-generation.It also includes unconditional protein structure generation.
  • 4.1 Task Taxonomy and Data Sources: Interaction tasks model molecule–protein and protein–protein relationships through sequence-only and structure-grounded binding-affinity prediction and interaction modeling.These tasks extend the corpus beyond single-entity biological inputs.
  • 4.1 Task Taxonomy and Data Sources: MegaScience supplies general scientific reasoning supervision to preserve Qwen3-inherited reasoning and instruction-following capabilities amid domain-specific biomolecular training.It is included during instruction tuning but excluded from the evaluation suite.
  • 4.2 Data Format Unification and Prompt Diversification: Datasets are standardized by serializing biomolecular fields with modality-specific boundary tokens and converting raw examples into instruction, input, and response triplets.Classification labels become natural-language category names, while regression targets use numerical strings with task-appropriate units and precision.
  • 4.2 Data Format Unification and Prompt Diversification: Prompt pools vary wording, sentence structure, and formality for each sub-task, reducing surface-form overfitting and improving robustness to paraphrased or unseen instructions.Each training instance samples one semantically equivalent template while preserving task intent.
  • 4.3 Training Configuration: Both BioMatrix-1.7B and BioMatrix-4B are tuned with AdamW, a 5.0 × 10^-5 peak learning rate, cosine decay, 10% warmup, sample-level masking, and 2,048-token sequences.Training uses the LLaMA-Factory framework.

5 Evaluation Setup

The evaluation tests whether BioMatrix’s unified tokenization and backbone support high-quality multimodal biomolecular tasks while remaining competitive with task-specialized models. It covers molecular, protein, and interaction tasks using task-group-specific instruction-tuned variants and benchmark-standard evaluation procedures.

  • Evaluation Questions: The evaluation asks whether biomolecular structures can share tokens with sequences and natural language under one next-token objective while supporting structural generation, prediction, and understanding.This feasibility question spans both molecules and proteins.
  • Evaluation Questions: The second question tests whether one unified backbone can match or surpass purpose-built or purpose-trained models across molecules, proteins, and their interactions.The comparison spans sequence, structure, and text within one vocabulary.
  • Task Organization: The benchmark taxonomy covers single-entity molecular tasks, single-entity protein tasks, and molecule–protein and protein–protein interaction tasks.Molecular and protein tasks are each organized across sequence and structure settings.
  • Evaluated Variants: Every reported result uses BioMatrix’s single tokenization scheme, continually-pretrained backbone, and next-token objective, while instruction tuning uses task-group-specific variants.The evaluated variants are instruction-tuned from the same continually-pretrained backbones and use the same unified tokenization scheme.
  • Baselines: Baselines include generalist biological foundation models such as BioT5+, NatureLM, SciReasoner, 3D-MoLM, ESM3, and ProtTeX.These models represent shared-framework comparisons for assessing progress on the multi-task frontier.
  • Evaluation Protocol: Each sub-task follows its source benchmark’s official data split and evaluation metric, with inference performed using vLLM.Tasks without output-diversity requirements use greedy decoding with temperature 0 for deterministic predictions.

6 Molecule Tasks

This section evaluates BioMatrix on molecular tasks spanning sequence-based and structure-based settings, including line notations, natural language, and molecular geometry. The suite covers both understanding and generation across these modalities.

  • Scope: BioMatrix’s molecular evaluation spans sequence-based tasks with molecular line notations and natural language, alongside structure-based tasks involving molecular geometry.Section 6.1 covers line notations with natural language, while Section 6.2 covers molecular geometry.
  • Sequence-based tasks: Sequence tasks include unconditional generation, name conversion, molecule editing, property prediction, captioning, text-based generation, customized generation, and molecule question answering.The listed mappings include no condition to valid SMILES / SELFIES, names or formulas to sequences, instructions to edited molecules, and molecule contexts to text answers.
  • Structure-based tasks: Structure tasks include forward and retrosynthesis, unconditional structure generation, and conditional structure generation from target properties.These tasks map reactants to products, products to reactants, no condition to a molecule’s 3D structure, or target properties to a 3D structure.

6.1 1D Molecular Tasks

BioMatrix performs strongly across 1D molecular understanding and generation tasks, often matching or surpassing specialized and larger baselines. Performance depends on representation: SMILES excels under strict structural constraints, while SELFIES benefits property optimization.

  • Molecular generation: BioMatrix-4B (SELFIES) reaches 0.998 validity, 1.000 uniqueness, and 0.986 novelty on GuacaMol, jointly outperforming OMG-GPT.Across both benchmarks, validity reaches 0.951–0.999 and uniqueness is at or near 1.0.
  • Molecular generation: BioMatrix maintains validity ≥0.951, uniqueness ≈1.0, novelty ≥0.891, and IntDiv1/2 above the ∼0.855 baseline, avoiding Sc2Mol’s validity–novelty trade-off.Sc2Mol reaches novelty 0.986 and IntDiv2 = 0.872 but validity collapses to 0.631.
  • Cross-modal translation: BioMatrix-4B improves I2S EM from SciReasoner-8B’s 84.40% to 92.83%, while BioMatrix-1.7B surpasses SciReasoner-1.7B on all EM metrics.BioMatrix-4B achieves the best result on four of five metrics and remains competitive on S2I EM.
  • Text-based generation: BioMatrix achieves the best result among LLM-based methods on all four text-generation metrics, raising EM from 48.00% to 65.07% and FTS from 77.50% to 85.47%.BioMatrix-1.7B already outperforms every LLM baseline on all four metrics.
  • Synthesis: BioMatrix-4B (SMILES) reaches 77.94% forward synthesis EM and 45.16% retrosynthesis EM, achieving the best or second-best LLM results across six metrics.SMILES variants consistently outperform SELFIES variants on EM; forward synthesis exceeds SciReasoner-8B’s 75.00%.
  • Customized generation: BioMatrix establishes state-of-the-art AddComp and DelComp results, including SR 0.9294 and WSR 0.6447 on AddComp, but trails Claude-3.5 on SubComp.On DelComp, BioMatrix-4B (SELFIES) attains SR 0.9666, while all variants exceed the prior best across SR and WSR.
  • Customized generation: BioMatrix-4B (SMILES) achieves AtomNum SR 0.2774 and BondNum SR 0.1104, but FuncGroup remains led by Llama3.1-8B at SR 0.3548.SELFIES performs dramatically worse on constrained tasks, with FuncGroup SR 0.0048 versus 0.2714 for SMILES.
  • Property prediction and optimization: BioMatrix establishes state-of-the-art logP and MR results across both metrics and remains competitive on QED, while SELFIES variants dominate property-optimization success.BioMatrix-4B (SELFIES) leads most Correct and Validity metric–task combinations across all eight MolOpt-Instructions sub-tasks.

6.2 3D Molecular Tasks

BioMatrix performs strongly on unconditional 3D molecule generation, matching or exceeding baselines on most chemical-validity and distributional metrics while showing weaker raw bond-length fidelity. Under property conditioning, it achieves the best or second-best MAE across all six QM9 targets, with especially large gains on electronic-structure properties.

  • Unconditional 3D molecule generation: BioMatrix-1.7B achieves FCD 0.064 versus 0.063 for the training reference and 0.070 for NExT-Mol, with strong chemical-validity scores.Frag and Scaf are 0.991 versus 0.992 and 0.944 versus 0.946 for the training distribution; atom stability, molecule stability, and V&C are 1.000, 0.988, and 1.000.
  • Unconditional 3D molecule generation: BioMatrix trails NExT-Mol on V&U&N, 0.749 versus 0.802, and SNN, 0.495 versus 0.530, indicating weaker novelty and similarity performance.These are the two reported 2D axes where BioMatrix trails NExT-Mol.
  • Unconditional 3D molecule generation: Raw BioMatrix conformers match diffusion baselines on dihedral-angle MMD but have bond-length MMD 1.05 and atom stability 0.897, reflecting looser geometric fidelity.The bond-length discrepancy is attributed to finite-codebook quantization error and per-atom error accumulation during autoregressive traversal.
  • Conditional 3D molecule generation: BioMatrix-4B attains the best MAE on five of six conditioned QM9 properties and the second-best on the remaining target, while BioMatrix-1.7B surpasses every baseline on all six.On electronic-structure targets, BioMatrix-4B reduces NExT-Mol’s MAE from 205 to 53 meV for εHOMO, 235 to 54 meV for εLUMO, and 297 to 81 meV for ∆ε.

7 Protein Tasks

BioMatrix is evaluated on protein tasks spanning sequence-based and structure-based settings, including sequence–language tasks and backbone-geometry tasks. Across protein understanding, it achieves parameter-efficient state-of-the-art performance against substantially larger baselines.

  • Protein task suite: Protein evaluation spans sequence-based and structure-based settings, covering amino acid sequences with natural language and tasks involving backbone geometry.Section 7.1 addresses sequence–language tasks, while Section 7.2 addresses backbone geometry.
  • Protein understanding: BioMatrix consistently matches or exceeds substantially larger baselines across sequence understanding, annotation prediction, and knowledge mining.This holds at both the 1.7B and 4B scales, including comparisons with 8B-scale SciReasoner and OPI-Llama variants.
  • Protein understanding: GPT-O3 and DeepSeek-R1 fail to generalize to the evaluated protein domains.The passage contrasts these frontier general-purpose LLMs with BioMatrix’s performance across protein understanding tasks.

7.1 1D Protein Tasks

BioMatrix performs strongly across protein-sequence understanding, annotation, and knowledge-mining tasks, substantially outperforming general-purpose and larger baseline models. It also achieves state-of-the-art fold-type results and high-quality text-conditioned protein generation.

  • Protein Sequence Understanding: BioMatrix-1.7B achieves 34.34% F1 on CLEAN EC Number Price, surpassing SciReasoner-1.7B at 10.00% and SciReasoner-8B at 22.00%.
  • Protein Sequence Understanding: BioMatrix-4B reaches 87.25% Family-level Accuracy and 30.22% Superfamily-level Accuracy, while BioMatrix-1.7B attains the highest Fold-level Accuracy at 16.71%.
  • Protein Annotation: BioMatrix-4B achieves 91.26% Keyword F1 and 86.33% GO term F1 on UniProtSeq, whereas GPT-O3 and DeepSeek-R1 show near-zero F1 on categorical annotation tasks.
  • Knowledge Mining: BioMatrix-1.7B achieves 41.64% Precision on gSymbol2Tissue, outperforming SciReasoner-8B at 41.00% and OPI-Llama-3.1-8B at 40.02%.
  • Text-Conditioned Protein Design: BioMatrix generates diverse, low-redundancy, structurally plausible proteins, with pLDDT values exceeding 60 on CAMEO and approaching 70 on MolInst-SwissProtCLAP.

7.2 3D Protein Tasks

BioMatrix performs strongly across 3D protein understanding, folding and inverse folding, and joint or unconditional generation. It achieves state-of-the-art or competitive results while using a shared vocabulary and autoregressive next-token decoding.

  • Protein structure understanding: On PFUD, BioMatrix-4B sets a new state-of-the-art across all four metrics, while BioMatrix-1.7B remains competitive and surpasses ProtTeXLlama3 on BLEU-2 and ROUGE-2.ProtTeXLlama3 is a larger Llama3-8B model incorporating structural tokens.
  • Protein folding: BioMatrix-1.7B and BioMatrix-4B both achieve TM = 0.84, with RMSD = 5.49 and RMSD = 5.38 respectively, surpassing smaller DPLM-2 models and closely tracking DPLM-2-3B.The models approach DPLM-2-3B while using a shared vocabulary for molecules, proteins, and text.
  • Inverse folding: BioMatrix-4B reaches 75.50% AAR and BioMatrix-1.7B reaches 75.20%, achieving the highest amino acid recovery through residue-aligned structure and sequence tokens.The residue-aligned vocabulary turns inverse folding into a token-to-token mapping problem.
  • Sequence–structure co-generation: In sequence-to-structure co-generation, both BioMatrix variants reach scTM = 0.965, with scRMSD of 2.81 for 1.7B and 2.80 for 4B, versus 0.907 / 6.34 for DPLM-2-650M.In the structure-to-sequence direction, BioMatrix-1.7B attains scTM = 0.979 / scRMSD = 1.50.
  • Unconditional structure generation: BioMatrix-4B achieves scTM = 0.963 with scRMSD = 3.07 for unconditional backbone generation, placing it near the native-PDB upper bound of 0.969.Its scTM exceeds RFDiffusion, FrameDiff, FoldFlow, and DPLM-2-650M, while its scRMSD trails RFDiffusion.

8 Interaction Tasks

BioMatrix handles molecule–protein and protein–protein interaction tasks by jointly processing heterogeneous entities in one token stream under a unified next-token objective, without pair-specific architectural modules. It achieves state-of-the-art or competitive results across sequence-only and structure-grounded interaction benchmarks, including affinity prediction and protein–protein interaction tasks.

  • General interaction capability: BioMatrix achieves state-of-the-art or highly competitive interaction performance by placing heterogeneous biomolecular entities side-by-side in one token stream, without pair-specific architecture or fusion modules.This applies to molecule–protein and protein–protein tasks in both sequence-only and structure-grounded regimes.
  • Sequence-based molecule–protein affinity: 1.030 RMSE: BioMatrix-4B surpasses every BindingDB baseline and the prior literature SOTA of 1.340.On PDBBind/CASF-2016, both BioMatrix-1.7B and BioMatrix-4B achieve RMSE = 1.268, versus literature SOTA 1.181.
  • Sequence-based protein–protein interaction: 62.18% ACC on Yeast and 83.12% ACC on Human: BioMatrix-4B surpasses every non-pretrained baseline and remains competitive with ESM-1b.On PPI Affinity, BioMatrix-4B achieves RMSE = 2.350, better than all non-pretrained baselines and close to ProtBert at 2.195.
  • Structure-grounded molecule–protein affinity: RMSE = 1.260, Pearson = 0.737, and MAE = 0.972: BioMatrix-4B achieves the best result on three of four PDBBindv2020 metrics.BioMatrix-4B ranks second on Spearman at 0.712, while BioMatrix-1.7B achieves the best Spearman at 0.717.
  • Structure-grounded molecule–protein affinity: On PDBBindv2020, BioMatrix-4B outperforms TankBind, a specialist using trigonometry-aware cross-attention between ligand and pocket geometry.The model uses shared tokenization, with ligand SELFIES-aligned geometry tokens and protein per-residue structure tokens placed together.

9 Discussion

The discussion identifies task-group-specific tuning, notation-dependent strengths, and scale-dependent performance patterns as important practical distinctions. It also finds that unified tokenization is most valuable for cross-modal and cross-entity tasks, while fine-grained geometric accuracy remains a limitation.

  • Instruction tuning: Mixing heterogeneous biological tasks in one SFT run rarely yields mutual gains and can destabilize small-data tasks.The released all-tasks model is competitive on large-data sub-tasks but trails task-group variants on a non-trivial fraction of small-data ones.
  • Notation choice: SELFIES variants reach 0.995–0.999 validity on unconditional generation, whereas SMILES variants reach 0.951–0.994.SELFIES therefore helps when validity-by-construction matters, although generation-level edge cases leave residual validity gaps.
  • Notation choice: SMILES outperforms SELFIES on customized generation and forward or retrosynthesis tasks, including 77.94/45.16 EM versus 69.08/42.90.For BioMatrix-4B customized generation, FuncGroup SR falls from 0.2714 with SMILES to 0.0048 with SELFIES.
  • Model scaling: Scaling from 1.7B to 4B yields the largest gains on knowledge-intensive structured-generation tasks, including 79.45 →94.09 I2F EM for SMILES molecular name conversion.Other reported gains include 56.35 →65.07 EM for text-based molecule generation and 72.75 →77.94 forward-synthesis EM.
  • Unified tokenization: Unified tokenization chiefly enables cross-modal and cross-entity tasks under one next-token objective, rather than dramatically improving single-modality specialist benchmarks.BioMatrix matches or does not dramatically exceed dedicated specialists on single-modality, single-entity tasks, while retaining the unified architecture without bolt-on encoders or fusion modules.
  • Geometric fidelity: BioMatrix matches or surpasses diffusion specialists on distribution-level or backbone-topology evaluations, but retains a residual gap on fine-grained per-atom geometry.The discussion proposes geometric post-processing as a practical refinement and points to richer structural tokenization or non-autoregressive decoding as future directions.

10 Conclusion

BioMatrix is a multimodal foundation model that natively integrates sequences, structures, and natural language for molecules and proteins in one decoder-only architecture. It represents heterogeneous biomolecular modalities in a shared discrete token space and processes them uniformly with a single next-token prediction objective.

  • BioMatrix integrates sequences, structures, and natural language for both molecules and proteins within a single decoder-only architecture.
  • A shared discrete token space lets BioMatrix handle heterogeneous biomolecular inputs and outputs uniformly under one next-token prediction objective.
  • The architecture operates without external encoders, projection adapters, or modality-specific output heads.

11 Limitations

BioMatrix’s limitations concern structural representation, data overlap, domain specialization, SFT organization, and incomplete coverage of biomolecular entities. These constraints motivate future work on complexes, evaluation hygiene, general-purpose capabilities, unified tuning, and broader biological modalities.

  • Overall limitations: The authors identify these limitations as important constraints that point to concrete directions for future work.The limitations are acknowledged despite the breadth and consistency of the empirical results.
  • Structural representation: BioMatrix uses disjoint token spaces for molecular and protein 3D structures, preventing native representation or generation of biomolecular complexes.The independent tokenizers cannot express the relative pose between a small molecule and a protein pocket.
  • Data overlap: No dedicated entity-level filtering was performed between continual pretraining and downstream SFT/evaluation data, allowing potential overlap through databases, literature, or cross-entity resources.Molecules, proteins, and associated annotations in downstream tasks may therefore have been implicitly observed during continual pretraining.
  • Domain specialization: The 304.4B-token corpus’s heavy biomolecular concentration shifts BioMatrix away from Qwen3-Base’s general language and reasoning capabilities despite mitigation efforts.The authors have not systematically benchmarked BioMatrix on general-purpose instruction-following or reasoning.
  • SFT organization: Task-group-specific SFT variants outperform the unified all-tasks SFT model, because mixed biological SFT data lets large-data subtasks overwhelm small-data ones.A single all-tasks model is released for practical use, but it still trails task-group variants on a non-...
  • Entity and modality coverage: BioMatrix currently covers small molecules and proteins but excludes DNA, RNA, carbohydrates, and lipids.Extending the unified tokenization scheme to these entities and modalities is presented as a natural scaling direction.

Appendix

The appendix documents BioMatrix’s instruction-tuning groupings and provides detailed training and test data statistics for each sub-task. The statistics continue across additional pages.

  • Table 30 presents the instruction-tuning grouping of BioMatrix.
  • Table 31 reports detailed training and test data statistics for each sub-task.
  • The detailed training and test data statistics continue from the previous page.
  • The appendix includes another continuation page for the detailed training and test data statistics.
Loading 2606.22138v1…