PaperlayerView exact revision on arXiv

Source-linked AI summary

Speaking the Language of Science: Toward a General-Purpose Generative Foundation Model for the Natural Sciences

Mingyang Li, Yurou Liu, Jieping Ye, Bing Su, Ji-Rong Wen, Zheng Wang

arXiv:2606.16905v1cs.CL

TL;DR

Scientific generation spans heterogeneous objects and constraints, but evidence for a unified modeling approach remains limited. LOGOS encodes these objects and interactions in a shared scientific grammar and, across six representative tasks, achieves highly competitive performance in a purely sequential paradigm, providing initial validation of the “one model fits all” premise.

  • Problem

    Scientific design requires generating novel objects under multiple constraints across heterogeneous domains, motivating evidence for a shared generative framework rather than modality-specific models.

  • Method

    LOGOS encodes scientific objects and spatial interactions as token sequences in a shared discrete scientific grammar, training one autoregressive model across domains and tasks.

  • Results

    Across six representative tasks, LOGOS achieves highly competitive performance within a purely sequential paradigm, providing initial validation of the “one model fits all” premise.

  • Takeaways & Limitations

    The results provide preliminary evidence that heterogeneous natural-science tasks can be addressed within one unified generative framework without explicit geometric dependencies.

  • Takeaways & Limitations

    CDR3, especially CDR-H3, is less tractable from sequence context alone and can benefit from explicit antibody three-dimensional structural constraints.

Abstract

from arXiv · show

In this report, we present LOGOS (Language Of Generative Objects in Science), a scientific generative language model that unifies heterogeneous tasks across the natural sciences within a single autoregressive framework based on a shared scientific grammar. It encodes diverse scientific objects and their spatial interactions as token sequences over a common vocabulary. By representing spatial contact and constraint patterns as discrete tokens, the model captures complex structural interactions in a purely sequential manner, without relying on explicit coordinates or geometric neural networks. This unified representation enables a wide range of downstream tasks to be formulated consistently as next-token prediction in the same grammar space, creating strong alignment between continued multi-domain pre-training and downstream objectives. Across diverse tasks, LOGOS consistently matches or outperforms domain-specific baselines, providing preliminary evidence for the feasibility of "one model fits all" in the natural sciences. We train LOGOS models at different scales (1B, 3B, and 8B parameters) and find a consistent positive correlation between model size and performance. This suggests that the future of AI for Science (AI4S) may not lie in building an independent technical stack that is separated from large language models (LLMs). Instead, it may depend on deeply aligning scientific foundation models with LLMs through shared architectures, shared training paradigms, and shared inference infrastructure, so that LLMs can truly become a new entry point for AI4S. We release the model weights and associated resources to facilitate further research.

1 Introduction

LOGOS proposes a unified autoregressive framework that represents heterogeneous scientific objects and their spatial interactions as token sequences under a shared scientific grammar. This design recasts understanding, prediction, and design tasks across natural-science domains as generative token modeling without explicit geometric dependencies.

  • Training and task formulation: LOGOS reformulates multi-domain scientific tasks as next-token prediction, bridging continued pre-training with downstream objectives under a shared representation.The reported task coverage spans proteins, antibodies, small molecules, chemical reactions, materials, and their interactions.
  • Motivation and framework: LOGOS unifies proteins, antibodies, small molecules, materials, reactions, and pockets as generative token-modeling problems within one grammar space.The framework advances beyond modality-specific specialized modeling toward general-purpose generative modeling, aligning training objectives with scientific applications.
  • Scientific grammar: The shared scientific grammar encodes diverse scientific objects and cross-object relationships in a common discrete token space.Spatial interactions are grammaticalized into tokens, enabling sequential modeling of structural information involving protein pockets, ligands, and their interactions.
  • Results: LOGOS achieves unified multi-domain, multi-task performance in a purely sequential paradigm without explicit geometric dependencies.Representative tasks include conditional ligand generation, materials generation and property-related tasks, retrosynthesis prediction, pocket mining, protein editing, and antibody CDR region prediction.
  • Resources: The authors release model weights, training-data processing pipelines, and related resources to support community research on general-purpose scientific foundation models.The release targets continued progress in unified representation, unified generation, and cross-domain scientific intelligence.

2 Method

LOGOS builds a unified scientific grammar that encodes seven scientific modalities and their interactions as discrete token sequences under a shared vocabulary. Nested pocket representations and interface tokenization connect protein sequence space with small-molecule chemical space.

  • Scientific grammar and corpus: LOGOS pretraining covers seven modalities spanning proteins, antibodies, small molecules, chemical reactions, materials, ligand-binding sites, and protein–ligand complexes.These modalities form biomacromolecular, chemical, and interaction-interface layers within the pretraining corpus.
  • Scientific grammar and corpus: All seven modalities are encoded as discrete token sequences under a shared vocabulary within one unified scientific grammar.The representation frames heterogeneous scientific objects and relationships in a common sequential format.
  • Data construction: The corpus applies modality-specific preprocessing, including filtering ambiguous antibody sequences, structural consistency checks for small molecules, and top-10 pocket selection using P2Rank.The antibody data come from OAS, small molecules from PubChemQC, materials from a hypothetical MOF database, and pockets from PDB structures.
  • Cross-modal representations: Pocket tokens are nested within protein grammar and can expand into small-molecule SMILES, making protein pockets a grammatical hub between protein and chemical spaces.This cross-level nesting links sequence-level protein representations with chemical representations.
  • Interaction encoding: Distance-based, multiradius residue selection discretizes continuous 3D proximity into interface tokens, while appended ligand SMILES records which residues bind which ligand.Together, these elements provide the tokenization of spatial interactions in the unified grammar.

3 Evaluation

LOGOS achieves strong or leading performance across ligand design, protein-pocket identification, retrosynthesis, and MOF generation within a unified sequential framework. Results also show parameter scaling, parameter efficiency, and sequence-only inference advantages across these tasks.

  • Ligand design: LOGOS-8B achieves the best Vina score of −7.76, while LOGOS-1B reaches −7.64 and outperforms all baseline models.LOGOS uses discretized token encoding of pocket residue features rather than explicit three-dimensional coordinates or dedicated geometric networks.
  • Ligand design: LOGOS-1B achieves a Vina score of −7.64 versus NatureLM (8×7B) at −6.91 with approximately 1/56 of the total parameters.The report attributes this efficiency to scientific grammar serving as a cross-modal interface that concentrates capacity on native scientific representation learning.
  • Scaling behavior: From 1B to 8B, LOGOS consistently improves binding affinity, drug-chemical properties, pocket identification, and retrosynthesis performance.For ligand design, Vina distributions shift toward stronger binding while QED and SA remain comparably favorable; pocket identification also benefits from multi-domain joint learning.
  • Pocket identification: LOGOS-8B ranks second only to P2Rank on Top-n and Top-(n+2) across COACH420 and HOLO4K, using only one-dimensional amino acid sequences.Unlike comparison methods requiring three-dimensional atomic coordinates, LOGOS enables pocket identification for proteins without resolved structures.
  • Retrosynthesis: LOGOS-8B achieves the best retrosynthesis Top-1 accuracy of 74.8%, outperforming NatureLM (8×7B) at 71.9% and other compared methods.Top-1 accuracy improves continuously from 64.0% to 71.5% to 74.8% as model size increases from 1B to 8B.
  • MOF generation: LOGOS significantly outperforms all domain-specific baselines across the three reported MOF-generation metrics and produces valid, structurally diverse crystalline frameworks.Several generated structures contain chemically plausible and geometrically coherent building blocks absent from the training set.

4 Conclusion and Future Work

LOGOS unifies heterogeneous scientific objects and spatial interactions in a shared discrete token space through a scientific grammar. Future work targets broader domain coverage, larger data and models, explicit geometry, and a general-purpose foundation model across domains, scales, and modalities.

  • Conclusion: LOGOS encodes proteins, antibodies, small molecules, chemical reactions, materials, and spatial interactions in a shared discrete token space.It uses domain-native representations and a unified scientific grammar rather than natural language as an intermediary.
  • Future Work — Domain coverage: Nucleic acid modalities, including genomic and transcriptomic sequences, remain outside the current framework.Extending the scientific grammar to these domains is identified as an important next step for broader natural-science coverage.
  • Future Work — Data and model scale: 1B to 8B parameter experiments show stable scaling behavior, while the pre-training corpus covers only a subset of publicly available domain data.Scaling both data and model size would test the framework’s capacity boundaries and characterize scaling on scientific modalities.
  • Future Work — Information modality: LOGOS currently represents spatial relationships through discretized tokens, leaving explicit geometric information as a potential complement for three-dimensional-structure-sensitive tasks.The proposed direction combines geometric information with sequential modeling.
  • Future Work — Long-term goal: These directions extend the feasibility study toward a general-purpose scientific foundation model for unified understanding, prediction, and design across domains, scales, and modalities.The long-term goal broadens the framework beyond its current scope.

Appendix · A Overview of data construction

LOGOS constructs training data as serialized scientific objects and relationships across multiple domains under a unified scientific grammar. Protein, small-molecule, and complex representations are linked through sequence transformations and discretized spatial interactions.

  • A Overview of data construction: LOGOS organizes scientific objects from multiple domains together with their relationships rather than learning from isolated single-modality data.This organization is summarized schematically in Figure 10 under the unified scientific grammar.
  • A Overview of data construction: Protein ligand-binding sites use multiple sequence forms to capture complementary object-level information.These forms include positional annotation in protein sequences, side-chain chemical representations, and amino-acid-to-small-molecule transformations.
  • A Overview of data construction: The construction explicitly connects protein residue representations with small-molecule chemical representations.The connection is established through transformations between amino-acid-level and small-molecule-level forms.
  • A Overview of data construction: Protein–ligand complexes are serialized jointly to encode interface-level structural constraints and interaction patterns.The joint representations organize protein pockets and ligands together.
  • A Overview of data construction: Spatial interactions are discretized and incorporated into the unified scientific grammar.This allows spatial relationships to be represented within the serialized data format.
  • A Overview of data construction: The data construction covers serialized representations of proteins, antibodies, small molecules, and metal clusters.Figure 10 distinguishes these modalities with green, light green, orange, and red colored bars, respectively.

B Evaluation metrics for different tasks

This section introduces the evaluation metrics used for different tasks and provides a detailed account of them.

  • The section explains the evaluation metrics used for different tasks.
  • It provides a detailed introduction to the metrics used in task evaluation.
  • The evaluation framework is organized around metrics for different tasks.

B.1 Interaction-Aware Ligand Design for Binding Pockets

Generated ligands are evaluated across binding affinity, drug-likeness, synthetic accessibility, and physicochemical properties using five complementary metrics. These metrics capture docking performance, molecular suitability, synthesis difficulty, lipophilicity, and Lipinski-rule compliance.

  • Evaluation metrics: The evaluation reports Vina score, QED, SAS, LogP, and LPSK to assess generated-ligand quality across multiple dimensions.The dimensions include binding affinity, drug-likeness, synthetic accessibility, and physicochemical properties.
  • Evaluation metrics: Vina score estimates binding free energy between a generated ligand and target protein pocket, with lower values indicating stronger predicted affinity.The score is derived from AutoDock Vina and averaged across multiple sampled ligands per method.
  • Evaluation metrics: QED measures drug-likeness on a 0-to-1 scale, with higher values indicating greater similarity to known oral drugs.It combines eight physicochemical properties, including molecular weight, logP, hydrogen bonding, polar surface area, rotatable bonds, aromatic rings, and formal charge.
  • Evaluation metrics: SAS estimates molecular synthesis difficulty from fragment complexity and rarity, ranging from approximately 1 for easiest to 10 for most difficult compounds.Scores below 6 are generally considered synthetically feasible, and the scale is inverted for consistency with metrics where higher is better.
  • Evaluation metrics: LogP measures octanol–water partitioning and reflects lipophilicity, with orally bioavailable candidates typically falling between -0.4 and 5.6.Values outside this interval may indicate poor absorption or excessive accumulation in fatty tissues; no directional preference applies within the window.
  • Evaluation metrics: LPSK measures the proportion of generated molecules satisfying Lipinski’s Rule of Five, with higher values indicating more favorable drug-like properties.The rule considers molecular weight, hydrogen bond donors, hydrogen bond acceptors, and LogP.

B.2 Protein Ligand-Binding Site Identification

Protein ligand-binding site identification uses ranking-based Top-n and Top-(n+2) metrics under the DCC criterion. A predicted pocket is correct when its center lies within 4 Å of any atom of a relevant ligand, with prediction counts adjusted for structures containing multiple relevant ligands.

  • Top-n and Top-(n+2) are ranking-based metrics for binding site identification.
  • Under the DCC criterion, a predicted pocket is correct if its center is within 4 Å of any atom of a relevant ligand.
  • When a protein structure contains multiple relevant ligands, the number of considered predictions is adjusted accordingly.

B.3 Pocket Translation Task

The pocket translation task evaluates correctness at both the residue-position level and the whole-pocket sample level, using strict translation criteria.

  • Site Accuracy: Site accuracy measures the proportion of amino acid residue positions translated correctly across all test-set samples.Each residue position is independently judged according to the stated correctness criterion.
  • Sample Accuracy: Sample accuracy measures the proportion of test-set pockets in which every residue position is correctly translated.A single mismatched, missing, or unparseable residue position makes the entire pocket sample incorrect.

B.4 Retrosynthesis Prediction

The section evaluates retrosynthesis prediction using Top-k, which measures whether the ground-truth reactant set appears among a model’s k highest-ranked candidates. This metric captures both prediction accuracy and ranking quality.

  • Top-k: Top-k measures the proportion of test examples whose correct reactant set appears within the top k predictions.For each target product, the model generates a ranked list of candidate reactant sets.
  • Top-k: A retrosynthesis prediction is correct when the ground-truth reactant set appears among the model’s top k candidate reactant sets.The candidates are ranked for each target product.
  • Top-k: Top-k reflects both prediction accuracy and ranking quality.It evaluates successful recovery of the correct reactant set within a specified candidate-list cutoff.

B.5 Unconditional Material Generation · B.6 Protein Editing

The sections define evaluation metrics for unconditional MOF generation and protein editing. MOF metrics progress from chemical validity to structural novelty and component innovation, while protein-editing metrics assess fitness, diversity, and departure from starting sequences.

  • B.5 Unconditional Material Generation: B.5 Unconditional Material Generation evaluates MOF generation with Valid, VNU, and NBB under the MOFFlow-2 protocol.These metrics assess increasingly strict aspects of generation quality.
  • B.5 Unconditional Material Generation: Valid is the proportion of generated MOFs passing MOFChecker’s chemical-validity check.Higher values indicate better performance.
  • B.5 Unconditional Material Generation: VNU is the proportion of generated MOFs that are chemically valid, structurally novel, and structurally unique after deduplication.Novelty requires the MOFid to be absent from the training set.
  • B.5 Unconditional Material Generation: NBB is the proportion of valid generated MOFs containing at least one novel building block absent from the training set.It measures component-level innovation.
  • B.6 Protein Editing: B.6 Protein Editing measures normalized fitness to quantify target-property improvement.Fitness reflects the extent to which the model improves the target property.
  • B.6 Protein Editing: Together, protein-editing metrics assess functional improvement, intra-set diversity, and departure from starting points.These provide complementary views of optimization quality.
  • B.6 Protein Editing: Diversity is the median pairwise edit distance among generated sequences, quantifying variation within the candidate set.It is computed over all pairwise comparisons.
  • B.6 Protein Editing: Novelty is the median minimum edit distance from each generated sequence to all starting sequences, measuring sequence-level departure.It quantifies how far generated sequences deviate from input sequences.

B.7 Antibody CDR Design

The antibody CDR design evaluation uses the standard SAbDab training/test split and assesses generated sequences with sequence-recovery, structural-consistency, and naturalness metrics. Results are reported separately for heavy- and light-chain CDRs.

  • Evaluation uses the established SAbDab benchmark with its standard training/test split for fine-tuning and testing.
  • The three widely used metrics are reported separately for heavy-chain CDR-H1/H2/H3 and light-chain CDR-L1/L2/L3.
  • Amino Acid Recovery (AAR, %) measures position-wise agreement between generated and native CDR sequences, with higher values indicating better recovery.
  • Self-consistency RMSD (scRMSD, Å) evaluates CDR structural compatibility after refolding with ABodyBuilder2, where lower values are better.
  • Plausibility uses AntiBERTy pseudo-log-likelihood to measure antibody-sequence naturalness, with higher values indicating greater consistency with natural antibodies.

C Detailed configurations

LOGOS uses unified continual pre-training and post-training across scientific domains, model scales, and backbones, with task-specific inference settings. Training runs use 32 NVIDIA A800 GPUs for pre-training and 8 for post-training.

  • Training configurations: Unified continual pre-training and post-training span multi-domain scientific data, different parameter scales, and different backbones.Detailed configurations are provided for these training settings.
  • Training configurations: 32 NVIDIA A800 GPUs are used for LOGOS continual pre-training.This configuration is specified in Table 9.
  • Training configurations: 8 NVIDIA A800 GPUs are used for LOGOS post-training.This configuration is specified in Table 10.
  • Inference configurations: Inference uses top_p 0.85, temperature 1.2, and repetition_penalty 1.05 for ligand design, retrosynthesis, and unconditional material generation.These settings apply to interaction-aware ligand design for binding pockets, retrosynthesis prediction, and unconditional material generation.
  • Inference configurations: Inference uses top_p 0.9, temperature 1.2, and repetition_penalty 1.0 for protein binding-site identification, protein editing, and antibody CDR design.These settings apply to protein ligand-binding site identification, protein editing, and antibody CDR design.
Loading 2606.16905v1…