PaperlayerView exact revision on arXiv

Source-linked AI summary

Scientific Large Language Models: A Survey on Biological & Chemical Domains

Qiang Zhang, Keyang Ding, Tianwen Lyv, Xinda Wang, Qingyu Yin, Yiwen Zhang, Jing Yu, Yuhao Wang, Xiaotong Li, Zhuoyi Xiang, Kehua Feng, Xiang Zhuang, Zeyuan Wang, Ming Qin, Mengyao Zhang, Jinlu Zhang, Jiyu Cui, Tao Huang, Pengju Yan, Renjun Xu, Hongyang Chen, Xiaolin Li, Xiaohui Fan, Huabin Xing, Huajun Chen

arXiv:2401.14656v2cs.CL

TL;DR

A comprehensive review of scientific LLMs was lacking despite general LLMs’ difficulty with scientific data. This survey systematically examines biological and chemical Sci-LLMs across language types, architectures, capabilities, datasets, and evaluation, while identifying challenges and future directions. It provides a broad technical overview, but its scope is limited to selected languages and primarily Transformer-based models.

  • Problem

    Existing general LLMs often fail to properly handle scientific data, while Sci-LLM advances have largely remained within separate research domains without a comprehensive unifying review.

  • Method

    The survey systematically reviews biological and chemical Sci-LLMs across textual, molecular, protein, genomic, and multimodal languages, comparing architectures, capabilities, training corpora, benchmarks, and evaluation criteria.

  • Results

    The survey presents a comprehensive overview of recent Sci-LLM advancements in biological and chemical fields and examines their multimodal interactions and evaluation.

  • Takeaways & Limitations

    The survey organizes technical developments and challenges across biological and chemical Sci-LLMs to support understanding of this emerging research area.

  • Takeaways & Limitations

    The survey excludes languages without universally defined vocabularies and grammatical structures and alternative neural architectures such as graph neural networks and diffusion models.

Abstract

from arXiv · show

Large Language Models (LLMs) have emerged as a transformative power in enhancing natural language comprehension, representing a significant stride toward artificial general intelligence. The application of LLMs extends beyond conventional linguistic boundaries, encompassing specialized linguistic systems developed within various scientific disciplines. This growing interest has led to the advent of scientific LLMs, a novel subclass specifically engineered for facilitating scientific discovery. As a burgeoning area in the community of AI for Science, scientific LLMs warrant comprehensive exploration. However, a systematic and up-to-date survey introducing them is currently lacking. In this paper, we endeavor to methodically delineate the concept of "scientific language", whilst providing a thorough review of the latest advancements in scientific LLMs. Given the expansive realm of scientific disciplines, our analysis adopts a focused lens, concentrating on the biological and chemical domains. This includes an in-depth examination of LLMs for textual knowledge, small molecules, macromolecular proteins, genomic sequences, and their combinations, analyzing them in terms of model architectures, capabilities, datasets, and evaluation. Finally, we critically examine the prevailing challenges and point out promising research directions along with the advances of LLMs. By offering a comprehensive overview of technical developments in this field, this survey aspires to be an invaluable resource for researchers navigating the intricate landscape of scientific LLMs.

1 INTRODUCTION

Scientific languages use domain-specific vocabularies and grammatical rules that differ from natural language, limiting general LLMs on molecules, proteins, and related data. This survey reviews Sci-LLMs for biological and chemical languages, covering architectures, capabilities, training data, evaluation, and multimodal interactions.

  • Scientific languages: Scientific languages include textual, mathematical, chemical, protein, and genomic systems with specialized vocabularies and grammatical rules.The same character can carry different meanings across systems, such as C denoting cysteine in protein language and carbon in SMILES.
  • Motivation: General LLMs often fail to properly handle scientific data because scientific and natural languages differ semantically and grammatically.
  • Scientific LLMs: Sci-LLMs are customized for scientific domains and support molecular property prediction, drug design, retrosynthesis, protein structure and function prediction, and protein generation.
  • Survey scope: The survey addresses a lack of comprehensive review by systematically examining Sci-LLMs across biological and chemical languages.
  • Survey scope: Its coverage includes textual, molecular, protein, genomic, and multimodal language modeling, organized by architectures, capabilities, training corpora, benchmarks, and task criteria.
  • Boundaries: The survey is limited to chemical and biological languages with defined vocabularies and grammatical structures and primarily Transformer-based models.It excludes mathematical languages and alternative architectures such as graph neural networks and diffusion models.
  • Contributions: The survey contributes a comprehensive Sci-LLM review, an evolutionary tree, resource listings, and an overview of multimodal interactions among scientific languages.The authors describe the multimodal overview as the first comprehensive treatment of this topic to their knowledge.

2 BACKGROUND

Scientific languages use domain-specific vocabularies and representations for chemistry, proteins, genomes, and other scientific knowledge. This section surveys their sequence-based modeling and organizes Sci-LLMs by architecture.

  • Scientific languages: Scientific languages have specialized vocabularies whose terms can carry meanings distinct from natural-language usage.The survey notes that proficiency in one language system does not necessarily transfer to another.
  • Molecular language: Molecular languages represent chemical structures through SMILES, SELFIES, and InChI strings encoding atoms, bonds, connectivity, and related properties.SMILES uses ASCII strings, SELFIES uses predefined symbols and syntax rules, and InChI provides layered machine-readable chemical identifiers.
  • Protein language: Protein language represents amino-acid sequences using symbols for the 20 standard amino acids and supports computational analysis of protein characteristics and functions.The system uses one-letter or three-letter amino-acid codes and lacks a traditional natural-language grammar.
  • Genomic language: Genomic language encodes DNA and RNA with nucleotide bases whose combinations form genetic sequences and support analysis of inheritance and gene expression.DNA uses A, T, C, and G, whereas RNA replaces thymine with uracil; genomic language lacks a clear grammatical rule system.
  • Scope: The survey focuses on one-dimensional sequence modeling and excludes models based solely on graph or three-dimensional structural modalities.Molecules, proteins, and genomes can also be represented as two-dimensional graphs or three-dimensional structures.
  • Model architectures: Sci-LLM architectures are categorized as encoder-only, decoder-only, or encoder–decoder models, with encoder-only systems producing representations for downstream understanding tasks.Decoder-only systems generate or extend sequences, while encoder-only systems compress inputs into representations used for tasks such as classification or clustering.

3 TEXTUAL SCIENTIFIC LARGE LANGUAGE MODELS

This section surveys textual scientific LLMs trained on textual corpora, emphasizing chemical and biological knowledge, their capabilities, datasets, and evaluation. It organizes models across medical, biological, chemical, and comprehensive scientific domains.

  • The survey reviews Text-Sci-LLMs trained on textual corpora, focusing on chemical and biological knowledge, capabilities, datasets, and evaluation.
  • Text-Sci-LLMs are organized into medical, biological, chemical, and comprehensive categories.
  • Medical Domain: Medical models are trained or adapted on clinical records, medical literature, textbooks, and dialogue datasets for question answering, prediction, and medical interaction.
  • Biological Domain: Biological models span encoder-only systems for biomedical NLP and decoder-only systems for generation, question answering, and relation extraction.
  • Chemical Domain: Chemical and materials models support chemical reaction extraction, compound-information mining, synthesis analysis, and materials-property extraction.
  • Comprehensive Domain: Comprehensive scientific models use broad scientific corpora and support scientific language understanding, retrieval, reasoning, numerical calculation, and knowledge-intensive tasks.

4 MOLECULAR LARGE LANGUAGE MODELS

This section surveys molecular language models for chemical discovery, organizing them by architecture and examining their capabilities, datasets, and evaluation criteria. It covers sequence-based, graph-aware, three-dimensional, and generative approaches for molecular representation, property prediction, and generation.

  • Overview: Mol-LLMs are reviewed through their architectures, capabilities, datasets, and evaluation criteria, with molecular generation receiving particular attention.The section presents a chapter overview and categorizes models by architecture.
  • Encoder-only Models: Sequence-based models can underrepresent molecular substructures, graph connectivity, and three-dimensional chemical information.The survey identifies limited attention to functional groups and the omission of graph and 3D structure as important constraints.
  • Molecular Representations: SMILES presents validity and robustness challenges, motivating SELFIES as an alternative molecular string representation.ChemBERTa processes both SMILES and SELFIES and examines representation size, tokenizer types, and string representations.
  • Encoder-only Models: BERT- and RoBERTa-based models use self-supervised learning over SMILES or SELFIES to learn molecular representations for downstream prediction tasks.Examples include masked SMILES recovery, multitask learning, molecular property prediction, and reaction fingerprint generation.
  • Graph and 3D Models: Graph- and 3D-integrated models combine Transformers or GNNs with structural information to improve molecular feature extraction and model spatial relationships.MG-BERT combines local message passing with BERT and atom-masking pretraining, while DeepSorption uses an SE(3)-equivalent Transformer for molecular crystals.
  • Decoder-only Models: Decoder-only GPT-based Mol-LLMs use SMILES to navigate chemical space and generate molecules with specified properties for drug discovery and materials science.MolGPT uses conditional training for property optimization, while SMILESGPT and iupacGPT are based on GPT-2.

5 PROTEIN LARGE LANGUAGE MODELS

The survey organizes Prot-LLMs by encoder-only, decoder-only, and encoder-decoder architectures, reviewing their representations, capabilities, datasets, and evaluation. Models increasingly incorporate evolutionary, structural, and functional information for protein understanding and generation.

  • Architectures: Prot-LLMs are classified into encoder-only, decoder-only, and encoder-decoder architectures suited to different protein research applications.Encoder-only models primarily support function or property prediction, whereas decoder-only models predominantly support protein generation.
  • Encoder-only Models: Encoder-only models use self-supervised objectives such as masked language modeling to learn protein sequence representations from large databases.Examples include ESM, ProteinBert, ProtTrans, PMLM, ProtFlash, and ProteinNPT, with methods targeting local, global, co-evolutionary, or sparse-label information.
  • Sequence and Structure Information: Multiple sequence alignment captures evolutionary dependencies but incurs substantial computational overhead and cannot handle orphan proteins.Retrieved Sequence Augmentation is presented as an alternative that links query sequences to structurally or functionally similar database sequences without extra alignment or preprocessing.
  • Sequence and Structure Information: Protein models increasingly integrate three-dimensional structure, combining sequence encoders, graph networks, structure-aware tokens, or multimodal generation.ESM3 integrates protein sequence, structure, and function and is described as adaptable for biological alignment tasks.
  • Knowledge and Data: Codon-trained language models provide high-quality protein representations that outperform comparable state-of-the-art models across a variety of tasks.Knowledge-enhanced approaches also incorporate Gene Ontology or contextual amino-acid information into protein representations.
  • Decoder-only Models: Decoder-only Prot-LLMs support controllable protein generation for protein engineering and drug design.ProGen uses conditioning tags for taxonomic, functional, and locational annotations, while ProtGPT2 generates sequences with compositions and disorder propensities comparable to natural proteins.

6 GENOMIC LARGE LANGUAGE MODELS

Gene-LLMs apply language-model architectures to DNA, RNA, and related genomic data, with reviews covering architectures, datasets, and evaluation. The surveyed models address limited annotations, cross-task transfer, long sequences, molecular-phenotype prediction, epigenomics, and genome evolution.

  • Overview: Gene-LLMs are reviewed by architecture, datasets, and evaluation for modeling genomic language across DNA and RNA sequences.The survey covers encoder-only, decoder-only, and encoder-decoder models, alongside convolutional and recurrent alternatives.
  • Encoder-only Models: Masked-sequence training enables genomic models to learn sequence patterns and support promoter, transcription-factor binding-site, and single-nucleotide variability prediction.MoDNA adds a Generator-Discriminator structure that integrates sequence and motif information during training.
  • Long-Sequence Modeling: Long-context genomic models address sequences of up to 36,000 base pairs using recurrent memory, while other systems use convolutional downsampling or long-range convolution.These designs target the challenge that genomic sequences often exceed natural-language lengths and make standard attention expensive.
  • Encoder-only Models: 50M to 2.5B-parameter transformer models provide transferable, context-specific nucleotide representations for molecular-phenotype prediction with limited data.They can also focus on regulatory elements such as enhancers without supervision and help prioritize functional genetic variants.
  • Functional Prediction: EpiGePT predicts genome-wide epigenomic signals using transcriptional-regulation context and learns cell-type-specific long-range interactions through self-attention.Uni-RNA uses a billion RNA sequences to extract evolutionary and structural information for RNA structure and function prediction.
  • Decoder-only Models: Decoder-only models support genomic generation and evolutionary analysis, including GenSLMs for SARS-CoV-2 evolutionary dynamics and DNAGPT for zero-shot genomic tasks.GenSLMs are reported to identify concerning variants precisely and rapidly, while DNAGPT supports species identification and regulatory-factor prediction.

7 MULTI-MODAL SCIENTIFIC LARGE LANGUAGE MODELS

MM-Sci-LLMs connect multiple scientific modalities, including molecules, proteins, genomes, and text, through architectures designed for cross-modal understanding and generation. The survey categorizes these systems and examines their capabilities and training and evaluation datasets.

  • Taxonomy: The survey categorizes MM-Sci-LLMs into molecule-text, protein-text, protein-molecule, and comprehensive models according to their focused modalities.Molecule-text systems include encoder, decoder, and encoder-decoder architectures for retrieval, prediction, translation, and generation.
  • Molecule-Text Models: Molecule-text models align molecular structures with textual descriptions using contrastive learning, cross-modal attention, graph encoders, and joint representation learning.These approaches support cross-modal retrieval, molecular property prediction, text-guided generation, and molecule-caption translation.
  • Molecule-Text Models: Decoder-based molecular models use graph-language integration, in-context few-shot learning, or instruction tuning for molecular generation and property prediction.DrugLLM targets desired pharmacochemical properties from limited examples, while MolecularGPT applies few-shot learning to molecular property prediction.
  • Encoder-Decoder Models: Encoder-decoder models unify molecular and textual representations for language, chemical, and cross-domain tasks, sometimes without task-specific fine-tuning.GIT-Mol additionally fuses SMILES, text, images, and structure graphs, while 3D-MolT5 integrates one-dimensional molecular sequences with three-dimensional structures.
  • Protein-Text Models: Protein-text models align protein sequences with biomedical text or translate textual descriptions into protein-function annotations and design tasks.ProtST combines unimodal mask prediction with multimodal representation alignment during protein pretraining.
  • Overview: MM-Sci-LLMs integrate molecules, proteins, genomes, and textual data, emphasizing interactions across modalities and the datasets used for training and evaluation.The survey focuses on cross-lingual multimodal models and excludes monolingual combinations such as joint protein-sequence and structure modeling.

8 CONCLUSION AND PERSPECTIVE

The survey reviews biological and chemical Sci-LLMs across scientific language types, architectures, capabilities, datasets, evaluation, and multimodal integration. It identifies data and cross-modal alignment challenges while outlining future directions involving richer representations, external knowledge, simulation, tools, and benchmarks.

  • Conclusion: The survey covers textual, molecular, protein, genomic, and multimodal Sci-LLMs, examining architectures, capabilities, datasets, and evaluation.
  • Conclusion: Generative Sci-LLMs receive particular attention because of their advantages and growing prominence in scientific applications.
  • Critical Challenges: Limited pre-training data constrains the development of larger Sci-LLMs, although sequencing advances are expanding available gene and protein sequences.
  • Critical Challenges: Fine-tuning effectiveness depends on dataset scale, diversity, balance, and cross-modal semantic alignment; label imbalance can favor majority-class predictions.
  • Future Directions: Future directions include larger high-quality cross-modal datasets, 3D stereo-temporal representations, external knowledge sources, physical simulation, specialized tools, and agents.
  • Future Directions: Integrating Sci-LLMs with external knowledge and molecular dynamics may reduce hallucination and improve interpretations of complex scientific phenomena.
  • Future Directions: Rapid and accurate evaluation metrics are needed to support quick benchmark construction and broader acceptance of Sci-LLM applications.

Contributions

The contributions section assigns responsibility for the survey’s sections among the listed authors.

  • Qiang Zhang, Keyan Ding, and Huajun Chen contributed to Sections 1, 2, and 8.
  • Sections 3 and 4 were contributed by teams including Jing Yu, Renjun Xu, Kehua Feng, Tao Huang, Pengju Yan, Xiaolin Li, Tianwen Lyu, Xiang Zhuang, Jiyu Cui, and Huabin Xing.
  • Sections 5–7 were contributed by teams including Yiwen Zhang, Zeyuan Wang, Zhuoyi Xiang, Hongyang Chen, Qingyu Yin, Mengyao Zhang, Jinlu Zhang, Xiaohui Fan, Xinda Wang, Xiaotong Li, Yuhao Wang, and Ming Qin.
Loading 2401.14656v2…