PaperlayerView exact revision on arXiv

Source-linked AI summary

Nature Language Model: Deciphering the Language of Nature for Scientific Discovery

Yingce Xia, Peiran Jin, Shufang Xie, Liang He, Chuan Cao, Renqian Luo, Guoqing Liu, Yue Wang, Zequn Liu, Yuan-Jyue Chen, Zekun Guo, Yeqi Bai, Pan Deng, Yaosen Min, Ziheng Lu, Hongxia Hao, Han Yang, Jielan Li, Chang Liu, Jia Zhang, Jianwei Zhu, Ran Bi, Kehan Wu, Wei Zhang, Kaiyuan Gao, Qizhi Pei, Qian Wang, Xixian Liu, Yanting Li, Houtian Zhu, Yeqing Lu, Mingqian Ma, Zun Wang, Tian Xie, Krzysztof Maziarz, Marwin Segler, Zhao Yang, Zilong Chen, Yu Shi, Shuxin Zheng, Lijun Wu, Chen Hu, Peggy Dai, Tie-Yan Liu, Haiguang Liu, Tao Qin

arXiv:2502.07527v3cs.AIcs.LG

TL;DR

Scientific foundation models are often trained in isolation, limiting integration across domains. NatureLM uses a sequence-based model trained across scientific fields and text; it matches or surpasses specialist models on several generation and translation tasks. The authors also report that language and few-shot capabilities remain areas for improvement.

  • Problem

    Scientific foundation models are often domain-specific, and other approaches face small scientific datasets or challenges aligning external modules with large language models.

    Source: §1 · p. 5

  • Method

    NatureLM applies a Transformer decoder to sequences representing scientific entities, training on a 143-billion-token corpus spanning scientific domains and text.

    Source: §1 · p. 6

  • Results

    NatureLM matches or surpasses state-of-the-art specialist models on generation and translation tasks, including retrosynthesis, SMILES-to-IUPAC translation, and protein generation.

    Source: §1 · p. 6

  • Takeaways & Limitations

    The authors present NatureLM as a generalist approach for scientific discovery across multiple domains.

    Source: §1 · p. 6

    NatureLM's language capabilities lag behind leading large language models, and it does not yet exhibit strong few-shot capabilities.

    Source: §11.2 · pp. 57–58

Abstract

from arXiv · show

Foundation models have revolutionized natural language processing and artificial intelligence, significantly enhancing how machines comprehend and generate human languages. Inspired by the success of these foundation models, researchers have developed foundation models for individual scientific domains, including small molecules, materials, proteins, DNA, RNA and even cells. However, these models are typically trained in isolation, lacking the ability to integrate across different scientific domains. Recognizing that entities within these domains can all be represented as sequences, which together form the "language of nature", we introduce Nature Language Model (NatureLM), a sequence-based science foundation model designed for scientific discovery. Pre-trained with data from multiple scientific domains, NatureLM offers a unified, versatile model that enables various applications including: (i) generating and optimizing small molecules, proteins, RNA, and materials using text instructions; (ii) cross-domain generation/design, such as protein-to-molecule and protein-to-RNA generation; and (iii) top performance across different domains, matching or surpassing state-of-the-art specialist models. NatureLM offers a promising generalist approach for various scientific tasks, including drug discovery (hit generation/optimization, ADMET optimization, synthesis), novel material design, and the development of therapeutic proteins or nucleotides. We have developed NatureLM models in different sizes (1 billion, 8 billion, and 46.7 billion parameters) and observed a clear improvement in performance as the model size increases.

1 Introduction

NatureLM is presented as a sequence-based model for scientific discovery, addressing the limits of isolated domain models with a shared approach across scientific fields. The authors report strong cross-task performance and gains as model size increases, while noting remaining language and few-shot limitations.

1 Introduction

  • Existing scientific models can be restricted to individual fields, while other approaches face small scientific datasets or limited alignment between external modules and large language models.Scientific models also struggle with precise quantitative predictions and rational design, and ensuring output accuracy remains a challenge.Source: §1 · pp. 5–6
  • NatureLM uses a sequence-based approach across biological, chemical, and material systems, leveraging sequence compatibility with language models and autoregressive modeling.It follows a Transformer decoder architecture and was trained on 143 billion tokens from scientific domains and text, followed by instruction tuning; reinforcement learning and dedicated fine-tuning are also available.Source: §1 · p. 6, Figure 1 · p. 7
  • NatureLM matches or surpasses state-of-the-art specialist models on generation and translation tasks including retrosynthesis, SMILES-to-IUPAC translation, and protein generation.The introduction also identifies material property prediction as an area of strong performance.Source: §1 · p. 6, Figure 1 · p. 7
  • 18 of 22 evaluated task categories improved with increasing model size, with 8x7B performing best, followed by 8B and then 1B.The 8x7B model has 46.7 billion total parameters and is a mixture-of-experts model.Source: §1 · p. 7, Figure 2 · p. 8
  • The authors report reinforcement learning gains for molecular property optimization and dedicated fine-tuning for retrosynthesis, while language and few-shot capabilities still lag behind leading large language models.They state that these limitations will be addressed in future iterations.Source: §1 · pp. 8–9

2 Method

NatureLM represents scientific entities as sequences and combines single-domain and cross-domain data to train a model spanning scientific domains. It then uses staged continued pre-training, instruction fine-tuning, and inference optimizations.

2.1 Pre-training data

  • Small molecules use SMILES, proteins and nucleic acids use FASTA, and materials are represented by composition and space-group sequences wrapped in entity-specific tokens.Reaction products and reactants and antibodies also receive dedicated tokens.Source: §2.1 · p. 10
  • Pre-training combines single-domain sequences with cross-domain data, including literature-interleaved entities, text-parallel scientific facts, and DNA segments linked to their translated proteins.Interleaved literature covers broader topics, while parallel databases provide more data for individual properties.Source: §2.1 · pp. 10–13

2.2 Post-training data

  • The post-training set contains about 5.1 million instruction-response pairs across six domains and more than 60 task categories, with manually varied prompts.The text data also draws on open-source instruction-tuning datasets to retain general language capabilities.Source: §2.2 · p. 13

2.3 Model architecture

  • The 8B and 8×7B models start from Llama 3 8B and Mixtral 8×7B, respectively; the 1B model uses a reduced-layer Llama 3 design and first trains on 300 billion SlimPajama text tokens.Scientific data is then used for continued pre-training across all three models.Source: §2.3 · p. 13

2.4 Continued pre-training

  • To reduce instability from newly added, randomly initialized scientific-token embeddings, continued pre-training first freezes existing parameters while training the new tokens, then trains the full network jointly.Validation loss is lower for larger models across all domains.Source: §2.4 · pp. 14–15, Figure 6 · p. 15

2.5 Post-training

  • Post-training mainly uses supervised fine-tuning on instruction-response pairs, computing loss only on responses and training one pair per sequence.The 1B and 8B models train for 20k steps and the 8×7B model for 7.8k steps because of resource constraints; RLHF after fine-tuning is also explored.Source: §2.5 · pp. 15–16

2.6 Inference acceleration

  • Inference uses PagedAttention and Selective Batching through vLLM; the 8×7B model reaches about 525 tokens per second at BF16 precision on two NVIDIA A100 GPUs.Source: §2.6 · p. 16

3 Small molecule tasks

NatureLM is evaluated across small-molecule generation, optimization, translation, and retrosynthesis tasks. Results include strong benchmark comparisons and improvements on property-conditioned generation and drug-design tasks.

3.1 Unconditional molecular generation

  • As model capacity grew, molecular validity and uniqueness increased, and NatureLM significantly outperformed generalist models on uniqueness.GPT-4 showed strong performance on validity.Source: §3.1 · p. 17, Table 3 · p. 17

3.2 Property-to-molecule generation

  • For TPSA-conditioned generation, Spearman correlation exceeded 0.8; generated molecules conditioned on QED 0.5 and TPSA 40 mostly centered near both requested values.Evaluations covered six properties, generating 100 molecules for each input value.Source: §3.2 · p. 18, Figure 7 · p. 18

3.3 Translation between SMILES and IUPAC

  • NatureLM (8x7B) outperformed competing language models on both SMILES-to-text and text-to-SMILES accuracy, performing comparably with specialist model STOUT.The evaluation used 2,993 pairs and reports no test-set leakage.Source: §3.3 · p. 19, Table 4 · p. 19

3.4 Target-aware hit compound generation and optimization

  • Larger NatureLM models generally achieved better docking scores in fragment generation conditioned on target proteins and retained molecular fragments.The section also tests generating compounds from target protein sequences for hit identification.Source: §3.4 · pp. 19–21, Figure 8 · p. 21

3.5 Text-guided binding affinity optimization

  • For 8 of 12 targets, over 50% of generated novel molecules improved docking scores; NatureLM also generated molecules with higher binding affinity than GPT4.Models 8B and 8x7B outperformed 1B on correct molecules for most targets.Source: §3.5 · p. 22

3.6 Text-guided metabolism & distributional property optimization

  • BBBP optimization success rates were 0.482, 0.549, and 0.552 for NatureLM 1B, 8B, and 8x7B, respectively; metabolism optimization success was highest for 8B.The authors note that the 8x7B model differs architecturally from the dense 1B and 8B models.Source: §3.6 · p. 23, Table 6 · p. 24

3.7 Retrosynthesis prediction

  • On USPTO-50K, all NatureLM sizes surpassed other methods in top-k accuracy; in a patent case, NatureLM predicted the ground-truth reactants while other methods needed refinement.The patent reaction was outside the training set.Source: §3.7 · pp. 24, 26, Table 7 · p. 25, Figure 10 · p. 25

4 Protein tasks

NatureLM is assessed on protein sequence generation, text-guided generation, antibody CDR-H3 design, protein annotation, and heme-binding protein design. Larger models show strong results across several of these tasks.

4.1 Unconditioned generation

  • For unconditioned generation, NatureLM (8x7B) achieved average length 318, diversity 0.989, and average pLDDT 75.9, outperforming Mixtral 8x7B and GPT-4 across these metrics.The pLDDT comparison was calculated on generated sequences shorter than 100 residues.Source: Table 8 · p. 27, §4.1 · p. 27

4.2 Text-guided protein generation

  • Larger models shifted stability and solubility score distributions toward the requested properties; all three models exceeded natural proteins on average stability scores.Two models also produced more sequences above the 0.5 stability threshold than natural proteins, while soluble and insoluble score distributions separated more with model size.Source: Figure 11 · p. 28, §4.2 · p. 29, Figure 12 · p. 29

4.3 Antigen-binding CDR-H3 design

  • NatureLM (8x7B) reached AAR 0.376 for CDR-H3 design, outperforming several specialized GNN models but not ABGNN; SFM-protein had the top performance.NatureLM uses sequence-only input for this task, while baselines may use additional information such as structure.Source: Table 10 · p. 30, §4.3 · p. 30

4.4 Protein characteristics description generation

  • NatureLM (8x7B) achieved the highest protein-annotation Rouge-L score, 0.585, and generated annotations that identified orthologues and protein functions.NatureLM models consistently outperformed the fine-tuned Llama 3 8B baseline, with differences widening as model size increased.Source: Table 11 · p. 31, §4.4 · p. 31

4.5 Heme-binding protein design driven by text and SMILES

  • Generated heme-binding candidates used text or heme-SMILES prompts, with histidine or methionine residues positioned to coordinate heme iron.The workflow generated protein descriptions, screened them for heme-related keywords, then predicted and inspected heme-bound structures; designed sequences differed significantly from database sequences.Source: §4.5 · pp. 31–34

5 Material tasks

NatureLM generates material compositions under unconditional, composition, or property instructions, while NatureLM-Mat3D converts formulas and space groups into crystal structures for evaluation. Results span improved material validity and stability with scaling, property targeting, and competitive structure prediction.

5 Material tasks

  • NatureLM generates material formulas and space groups, and the specialized NatureLM-Mat3D predicts crystal structures from those outputs.The structure predictor represents formulas, space groups, lattice parameters, and fractional coordinates as sequences; training uses 6.5 million deduplicated samples from Materials Project, NOMAD, and OQMD, with benchmark test data removed.Source: §5 · p. 35, §5.4 · p. 40

5.1 Unconditional material generation

  • Larger models improve SMACT validity and stability for unconditional material generation; stability is defined as ehull < 0.1 eV/atom.NatureLM-Mat3D predicts structures for a randomly selected subset of valid compositions, whose ehull is evaluated with MatterSim.Source: §5.1 · p. 35, Table 12 · p. 35

5.2 Material generation for given composition

  • For composition-conditioned generation, NatureLM produces materials containing specified elements; Table 13 evaluates validity, stability, precision, and novelty.The supplied passages describe the evaluation dimensions but do not report their results.Source: §5.2 · p. 36, Table 13 · p. 36

5.3 Material generation for desired properties

  • As NatureLM scales, bulk-modulus predictions align more closely with instructed values, while validity and stability improve.Two generated structures targeted at 400 GPa were validated by DFT at 390 and 394 GPa and were novel relative to Materials Project.Source: §5.3 · pp. 37–39, Table 14 · p. 38, Figure 17 · p. 39

5.4 NatureLM-Mat3D: a crystal structure predictor for materials

  • NatureLM-Mat3D (1B) matched or surpassed state-of-the-art methods, with better performance as atom counts increased.Its match rates and RMSE values indicate it captures crystal arrangements; Table 15 reports match rate and RMSE across Perov-5, MP-20, and MPTS-52.Source: §5.4 · pp. 40–41, Table 15 · p. 41

6 Nucleotide tasks

NatureLM is evaluated on unconditional RNA generation, guide RNA design, and protein-conditioned RNA generation. Larger models show stronger results on multiple RNA-generation measures, and the reported guide RNA and binding evaluations indicate task-specific capabilities.

6.1 Unconditional RNA generation

  • Larger models generated RNA with lower, more negative MFE values and retrieved more unique Rfam families: 23, 38, and 165 for 1B, 8B, and 8x7B.Shuffling or randomizing generated sequences increased MFE, suggesting their original sequences contain structural features associated with stability.Source: §6.1 · pp. 42–43, Table 16 · p. 43

6.2 Guide RNA design

  • NatureLM achieved 95.7% accuracy in designing valid guide RNAs and also selected more effective candidates from pairs.Valid crRNAs must be 17–24 nucleotides, match a region in the provided DNA, and be followed by an NGG PAM.Source: §6.2 · p. 44, Table 17 · p. 44

6.3 Protein binding RNA design

  • For protein-conditioned RNA, larger NatureLM models performed better; generated sequences were more likely to be predicted to bind their specified proteins than unconditioned sequences.Predicted binding RNA averaged 0.714 with a 74.5% success rate, versus 0.391 and 36.3% for unconditioned sequences and 0.274 and 24.4% for non-binding controls.Source: §6.3 · p. 45, Table 18 · p. 46

7 Prediction tasks

NatureLM is evaluated on small-molecule, protein, and DNA property-prediction tasks. Larger models generally perform better, with competitive results on protein tasks but remaining gaps to specialist models on some evaluations.

7.1 Small molecule prediction tasks

  • Larger NatureLM models generally achieve better AUROC on MoleculeNet BBBP, BACE, and Tox21 tasks, but remain behind state-of-the-art specialist models.The tasks cover blood-brain-barrier crossing, BACE receptor binding, and toxicity associated with 12 targets.Source: §7.1 · p. 47, Table 19 · p. 47

7.2 Protein prediction tasks

  • NatureLM models perform competitively with task-specific protein models and surpass them on some tasks, including stability prediction.The same unified models are evaluated separately across tasks, whereas specialist models are trained and tested individually for each task.Source: §7.2 · p. 48, Table 20 · p. 48

7.3 DNA prediction tasks

  • On DNA promoter and transcription-factor-binding tasks, performance improved with model size but remained below state-of-the-art.Transcription-factor-binding performance is averaged across five specified ChIP-seq datasets and evaluated using MCC.Source: §7.3 · p. 49, Table 21 · p. 49

8 Strategies to further improve performance

The paper tests reinforcement learning and dedicated fine-tuning as strategies for improving NatureLM. The reported outcomes include improvements across molecular optimization tasks, competitive retrosynthesis, and state-of-the-art results on two Matbench tasks.

8.1 Reinforcement enhanced NatureLM

  • DPO improved performance across nine molecular property-optimization tasks without introducing new data during reinforcement training.The model was already instruction-tuned; generating data on the fly and using additional reward models are described as future plans.Source: §8.1 · p. 51, Table 22 · p. 51

8.2 Dedicated fine-tuning on retrosynthesis

  • NatureLM 1B performed competitively with leading template-based and template-free models on Pistachio retrosynthesis.The fine-tuning used a cleaned reaction dataset, root-aligned SMILES, and tenfold augmentation of the training data.Source: §8.2 · pp. 51–52, Table 23 · p. 52

8.3 Dedicated fine-tuning on Matbench

  • A single fine-tuned NatureLM 8B model achieved state-of-the-art performance on Matbench expt gap and matbench is metal.The model was fine-tuned jointly for three Matbench tasks rather than as separate task-specific models.Source: §8.3 · p. 52, Table 24 · p. 52, Table 25 · p. 52

9 Text capabilities

NatureLM’s case studies examine scientific language capabilities in CRISPR gene editing, cancer immunotherapy, and drug discovery, followed by an AlpacaEval comparison with Mixtral. The examples show useful domain knowledge but also a peptide-length error, while benchmark results indicate constrained general-language performance.

9.1.1 CRISPR gene editing

  • NatureLM explains CRISPR tools’ advantages and disadvantages, including Cas9’s editing efficiency, targeting range, delivery challenges, and repair-control limits.The example also notes possible off-target effects and chromosomal abnormalities.Source: §9.1.1 · p. 53, Figure 18 · p. 53

9.1.2 Cancer immunotherapy

  • NatureLM explains checkpoint-inhibitor mechanisms and generates a valid PD-L1-inhibiting peptide sequence, but the peptide length is incorrect.The accompanying case is shown in Figure S18.Source: §9.1.2 · p. 54, Figure 19 · p. 54

9.1.3 Drug discovery

  • NatureLM provides mostly accurate answers about Lipinski’s Rule of Five and strategies for improving drug ADME properties.The case study frames drug discovery as combining empirical rules, computational design, and experimental validation.Source: §9.1.3 · p. 54, Figure 20 · p. 54

9.2 Comparison on AlpacaEval evaluation

  • 31.8% LC win rate and 25.2% win rate against Mixtral show NatureLM’s constrained general-language performance on AlpacaEval.The length-controlled metric controls for response-length bias.Source: §9.2 · p. 55

10 Ablation study

The ablations test general-text instruction data during post-training and scientific-data continued pre-training. Scientific pre-training improves performance across tasks, while the effect of general-text post-training varies with model size.

10 Ablation study

  • The ablations examine post-training text data and scientific continued pre-training across small-molecule, protein, and DNA tasks.The study compares NatureLM models with and without text-based post-training data and a directly fine-tuned Llama 3 8B baseline.Source: §10 · p. 56, §10.1 · p. 56

10.1 Impact of text-based post-training data

  • For 1B models, omitting general-text instruction data yields better scientific-task performance on 5 out of 8 tasks; for 8B models, including it does not hurt performance.The authors suggest limited capacity may dilute the smaller model’s focus, whereas 8B models can accommodate both text and science instructions.Source: Figure 21 · p. 56, §10.1 · p. 57

10.2 Impact of continued pre-training on scientific data

  • NatureLM 8B outperforms directly fine-tuned Llama 3 8B across all evaluated tasks, supporting the value of continued scientific pre-training.Figure 21 reports the ablation results.Source: Figure 21 · p. 56, §10.2 · p. 57

11 Discussions

The discussion presents NatureLM as a multi-domain sequence-based science model, reports limits in language and few-shot capabilities, and outlines possible cross-domain research directions. The proposed directions include tasks that combine material properties with protein interactions and integrate RNA, proteins, and small molecules.

11.1 Summary

  • NatureLM is a sequence-based science model spanning multiple domains, with larger models performing better on 18 of the 22 tested tasks.The paper describes text-instruction generation, cross-domain generation, and predictive capabilities across scientific entities.Source: §11.1 · p. 57

11.2 Limitations

  • NatureLM’s limitations include a 31.8% AlpacaEval win rate against Mixtral 8x7B and currently weak few-shot learning capabilities.The authors plan to improve language capabilities with more high-quality pre-training text and to refine training strategies and increase model size for few-shot learning.Source: §11.2 · pp. 57–58

11.3 Cross-domain applications

  • The authors propose cross-domain applications such as biocompatible-material design involving protein interactions and ribozyme or bio-catalyst development combining RNA, protein, and small-molecule knowledge.These are described as future work to study, not established applications.Source: §11.3 · p. 58

Author list

The author list identifies the paper’s contributors and corresponding authors. Supplementary materials provide additional figures and tables covering molecular and material generation, cross-domain examples, training details, and evaluation data.

Author list

  • Yingce Xia and Tao Qin are corresponding authors; the paper identifies its co-first authors with an asterisk.The work was conducted at Microsoft Research AI for Science.Source: Author list · p. 59

A Supplementary figures

  • Supplementary molecular figures cover property-conditioned generation, molecular-property distributions, retrieval and docking evaluations, and generated-material stability and novelty.The captions describe correct-ratio criteria, molecular property plots, energy-above-hull distributions, and novel materials relative to generated materials.Source: Figure S1 · p. 72, Figure S2 · p. 73, Figure S4 · p. 74
  • Additional examples show SMILES translation and retrosynthesis comparisons, heme-binding protein designs, and generated-protein sequence-length and RNA-score distributions.The captions also describe generated-protein structural comparisons and predicted RNA binding-affinity trends across model sizes.Source: Figure S9 · p. 78, Figure S10 · p. 79, Figure S11 · p. 80

B Supplementary tables

C.1 Text-guided basic property optimization of small molecule compounds

NatureLM optimizes molecular properties from text commands and SMILES inputs using a fine-grained procedure. Reported performance improves with model size, although DrugAssist* has the highest overall scores.

C.1 Text-guided basic property optimization of small molecule compounds

  • The optimization input pairs a text command with a SMILES sequence, and the evaluation covers QED, LogP, and hydrogen bond donor count.The procedure follows DrugAssist [107].Source: §C.1 · p. 92
  • NatureLM (8B) surpasses NatureLM (1B) in all categories, with performance metrics improving as model size increases; DrugAssist* achieves the highest overall scores.Table S7 reports success rate and uniqueness ratio for the basic property optimization comparison.Source: Table S7 · p. 92, §C.1 · pp. 92–93

C.2 Supplementary information of RNA generation

  • RNA supplementary information lists ViennaRNA RNAfold for minimum-free-energy calculation and a cmscan command using Rfam files.Source: §C.2 · p. 93

C.4 Supplementary information for evaluation metrics

The supplementary evaluation defines success criteria for BBBP and CYP optimization. The sequence-model discussion contrasts text-oriented scientific models with NatureLM's sequence-based, discovery-focused approach.

Success Rate for BBBP and CYP Optimization

  • BBBP optimization counts a compound as successful when BioT5 predicts it can cross the BBB; CYP optimization succeeds when the output's predicted inhibition value is lower than the input's.BBBP test compounds initially cannot cross the BBB; the CYP predictor uses 0 for inhibition and 1 for no inhibition.Source: Success Rate for BBBP and CYP… · pp. 93–94

C.5 Shift the focus from general text to scientific sequences

  • Table S8 compares NatureLM with existing sequence models; the discussion identifies cross-domain generation and optimization as NatureLM's discovery focus.Source: Table S8 · p. 94, §C.5 · p. 95
  • NatureLM uses 10% pure text and 90% scientific entities and cross-domain sequences, whereas Galactica uses over 90% pure text; NatureLM targets generative discovery tasks.The discussion contrasts sequence-based generation and optimization with scientific-text understanding and reasoning.Source: §C.5 · pp. 94–95
Loading 2502.07527v3…