A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery

• Scientific LLMs extend beyond text tasks to data such as molecules, proteins, tables, and metadata, and may support discovery through generation and planning.The paper describes both handling existing scientific data and using LLMs in discovery-related work.Source: §1 · p. 1
• Earlier surveys often covered only one or two fields or a single modality, while a holistic view reveals related techniques across fields and modalities.The paper frames its cross-field, cross-modal survey as a way to make those connections visible.Source: §1 · pp. 1–2
• The survey groups pre-training into masked language modeling on sequentialized data, next-token prediction on sequentialized data, and contrastive alignment of paired modalities.The first two strategies use encoder or (encoder-)decoder models; contrastive learning maps relevant data pairs closer in latent space.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The survey covers more than 260 scientific LLMs across fields, modalities, and model sizes, and summarizes their datasets, architectures, and evaluation tasks.It also organizes models in appendix tables and reviews applications spanning hypothesis generation, theorem proving, experiment design, drug discovery, and weather forecasting.Source: §1 · p. 2

2.1 Language

• Paper corpora from bibliographic databases are common; BERT-based models often use MLM, GPT-based models next-token prediction, and newer models may add instruction tuning.General scientific LLMs are commonly evaluated on NER, relation extraction, question answering, and classification.Source: §2.1 · p. 3

2.2 Language + Graph

• Graph-aware models combine paper text with metadata or use citation links as supervision, and are evaluated on link prediction, retrieval, recommendation, and author-name disambiguation.SciDocs and SciRepEval are widely used benchmarks.Source: §2.2 · p. 3

2.3 Applications in Scientific Discovery

• Applications include searching for scientific directions, generating ideas and hypotheses, matching reviewers to submissions, and providing feedback on research papers.The survey also notes increasing use of LLMs in scientific-paper and peer-review writing.Source: §2.3 · p. 3

3.1 Language

• Mathematics LLMs evolved from BERT models trained mostly with MLM to GPT- and LLaMA-based models using next-token prediction and instruction tuning.With enormous corpora such as mathematical web pages and code, next-token prediction remains a pre-training task or companion task; QA and math word problems are common evaluations.Source: §3.1 · pp. 3–4

3.2 Language + Vision

• Geometry LLMs encode images into linearized visual representations and use sequence-to-sequence training, taking a problem as input and producing an answer, optionally with rationale.Approaches include detecting image relationships before applying BART, or concatenating vision-transformer embeddings with text embeddings before feeding them to LLaMA-2; auxiliary image or text-image losses may be added.Source: §3.2 · p. 4

3.3 Table

• Table LLMs commonly place linearized table text alongside the question, using objectives such as MLM or instruction tuning; evaluation typically uses table question answering.TableInstruct integrates 14 datasets across 11 tasks.Source: §3.3 · p. 4

3.4 Applications in Scientific Discovery

• 27 out of 30 classical geometry problems were solved by an AlphaGeometry extension using Wu’s method, compared with AlphaGeometry’s 25 out of 30.FunSearch also found a new solution to the cap set problem; the passage reports it can be faster and more efficient than human-devised solutions.Source: §3.4 · p. 4
• LLMs can iteratively propose and critique statistical models using in-context learning and chain-of-thought reasoning.Source: §3.4 · p. 5

4.1 Language

• astroBERT is further pretrained on astronomy papers using MLM and next-sentence prediction, while AstroLLaMA is fine-tuned on over 300,000 astronomy abstracts.PhysBERT is trained on a corpus based on 1.2 million physics papers and evaluated on physics-oriented tasks including retrieval, classification, and semantic similarity.Source: §4.1 · p. 5

4.2 Applications in Scientific Discovery

• Transformer-based physics applications include predicting integer coefficients in planar N = 4 Super Yang-Mills scattering amplitudes, learning Rydberg-atom qubit measurement distributions, and generating experimental blueprints.The blueprint work is described as an initial trial using a code-generating LLM to produce Python for a whole class of quantum systems.Source: §4.2 · p. 5

5.1 Language

• Chemistry text models have shifted from moderate-sized encoder-only MLM systems to decoder-only models trained with next-token prediction and instruction tuning.Evaluation spans labeling and classification as well as name conversion, reaction prediction, retrosynthesis, molecule design, and crystal generation.Source: §5.1 · p. 5

5.2 Language + Graph

• Molecular graph-and-text systems either contrastively align GNN and LLM encoders or feed linearized or tokenized graphs into an LLM.Text2Mol supports text-to-molecule retrieval, while 3D-MoLM uses molecular tokens with instructions for molecule-to-text retrieval and captioning.Source: §5.2 · p. 5

5.3 Language + Vision

• Molecular vision models project images into tokens for LLM input, while GIT-Mol projects graphs and images into a latent text space for T5 encoding and decoding.The passage situates these approaches alongside the BLIP-2 philosophy of representing images as tokens.Source: §5.3 · pp. 5–6

5.4 Molecule

• Molecule-only models commonly represent molecules as SMILES or SELFIES, using bidirectional encoders for understanding and autoregressive or graph-aware Transformers for generation and prediction.Evaluations include molecular property prediction, virtual screening, molecule generation, reaction prediction, and retrosynthesis.Source: §5.4 · p. 6

5.5 Applications in Scientific Discovery

• LLMs support autonomous chemical research and tasks including drug and catalyst design, molecular discovery, and drug editing, while benchmarks also identify continuing safety and utility needs.Reported approaches include tool-integrating agents, interactive models, and in-context design without traditional training or simulation; fine-tuned LLMs perform comparably to or better than conventional techniques on many chemistry applications.Source: §5.5 · p. 6

6.1 Language

• Biomedical language models evolved from moderate-sized encoder and encoder-decoder systems to instruction-tuned billion-parameter LLMs, alongside bi-encoders targeting retrieval.Evaluations range from biomedical NER, relation extraction, similarity, classification, and QA to single-round QA and multi-round dialogue; retrieval benchmarks include NFCorpus and TREC-COVID.Source: §6.1 · p. 7

6.2 Language + Graph

• Biomedical graph methods use ontology signals with text encoders or linked-paper text as pretraining context, and both approaches improve QA requiring complex reasoning.The approaches include deeply fusing LLM and GNN representations and appending a linked paper as a metadata neighbor for MLM.Source: §6.2 · p. 7

6.3 Language + Vision

• Biomedical vision-language models commonly use CLIP-style contrastive alignment of text and image encoders, with LLaVA-Med instead feeding image-derived visual tokens to an LLM.Evaluation tasks include image classification, segmentation, detection, vision QA, cross-modal retrieval, and report generation.Source: §6.3 · p. 7

6.4 Protein, DNA, RNA, and Multiomics

• Protein, DNA, and RNA models use sequence encoders with MLM or decoders with next-token prediction, while protein-text models align sequences and descriptions or input protein tokens with text.Multiomics LLMs mainly address single-cell transcriptomics; Performer is widely used there because of its linear attention complexity on long scRNA-seq data.Source: §6.4 · pp. 7–8

6.5 Applications in Scientific Discovery

• LLMs are applied to biological and medical research automation, clinical evidence synthesis, protein structure prediction and design, antibody generation, and viral-variant fitness evaluation.The cited examples include CRISPR experiment design and extracting and synthesizing clinical evidence from literature.Source: §6.5 · p. 8

7.1 Language

• Geoscience LLMs scale from encoder-based MLM models to instruction-tuned autoregressive models evaluated across QA, classification, probing, reasoning, summarization, and generation.Examples include geolocation MLM for entity linking and POI matching, and domain-adapted models such as K2 and OceanGPT.Source: §7.1 · p. 8

7.2 Language + Graph

• Geoscience graph-and-text systems fuse POI information within a BERT-based architecture or combine an LLM with a knowledge-graph pointer network for dialogue.Source: §7.2 · p. 8

7.3 Language + Vision

• UrbanCLIP uses CLIP-style text-image contrastive learning with aerial views and location descriptions for urban indicator prediction.Source: §7.3 · p. 8

7.4 Climate Time Series

• Climate foundation models use ViT and Swin Transformer architectures pretrained on ERA5 and CMIP6 climate time series for weather forecasting.Source: §7.4 · p. 8

7.5 Applications in Scientific Discovery

• Reported applications include LLM potential for urban planning, earthquake detection and phase picking, and climate conversational AI supplied with external scientific knowledge.Urban planning perspectives include sustainability, living, economic, disaster, and environmental considerations.Source: §7.5 · pp. 8–9

Diving into Fine-Grained Themes

• General scientific pretraining may let frequent signals dominate model parameters and wipe out domain-specific tail knowledge, motivating theme-focused knowledge graphs to guide generation.Source: §8 · p. 9

Generalizing to Out-of-Distribution Scientific Data

• Scientific LLMs face testing-distribution shifts from newly emerging concepts and unseen molecules or proteins, and handling such out-of-distribution data remains a challenge.The survey identifies invariant learning as a possible theoretical foundation and its integration into pretraining as worth exploring.Source: §8 · p. 9

Facilitating Trustworthy Predictions

• Plausible but factually incorrect generations pose particular risks in high-stakes fields such as chemistry and biomedicine; cross-modal RAG is proposed as an opportunity to improve trustworthiness.Existing scientific RAG studies mainly retrieve text and knowledge, while cross-modal RAG could guide text generation with relevant chemicals and proteins.Source: §8 · p. 9

Limitations

• The survey primarily covers mathematics and natural sciences, leaving social-science applications for future work because of space limitations.The authors note social-science uses such as representative tasks and social-simulation experiments.Source: Limitations · p. 9
• Studies proposing scientific-problem benchmarks but evaluating only general-purpose LLMs are outside the survey’s scope.Source: Limitations · p. 9
• Models spanning multiple fields or modalities are introduced in only one subsection under the survey’s classification scheme, for brevity.The authors give BioMedGPT and GIT-Mol as examples of models with overlapping categories.Source: Limitations · p. 9

A Summary Tables of Scientific LLMs

• The tables record each model’s modality, parameter count, architecture, pre-training data and tasks, and evaluation tasks.Models are grouped by modality within each field and sorted chronologically; publication dates use the preprint date when available, otherwise the conference or journal date.Source: A Summary Tables of Scientific LLMs · p. 28
• The tables cover general science, mathematics, physics, chemistry and materials science, and biology and medicine.The next table is identified as covering geography, geology, and environmental science, but its model entries are not included in this part.Source: Table A1 · p. 28, Table A2 · p. 28, Table A3 · p. 29
• The listed models span language, language-plus-vision, table, molecule, protein, DNA, RNA, and multiomics modalities.Their listed evaluation tasks include question answering, problem solving, molecule and protein generation or prediction, and biomedical image tasks.Source: A Summary Tables of Scientific LLMs · pp. 29, 30, 33, 34
• ClimateBERT-NetZero, GeoLM, and GeoGalactica represent geoscience models ranging from 82M to 30B parameters [Schimanski et al., 2023; Li et al., 2023f; Lin et al., 2024c].ClimateBERT-NetZero uses DistilRoBERTa for Net Zero Tracker classification; GeoLM uses BERT and contrastive learning with OpenStreetMap and Wikipedia for entity tasks; GeoGalactica uses Galactica, geoscience papers, code, Wikipedia, and instruction tuning for reasoning, summarization, and generation.Source: A Summary Tables of Scientific LLMs · p. 35
• ERNIE-GeoL combines a Transformer with Baidu Maps and graph aggregation for address parsing, geocoding, and query–POI matching [Huang et al., 2022].Its listed data sources include a POI database, geocoding, and search logs.Source: A Summary Tables of Scientific LLMs · p. 35
• UrbanCLIP combines a Transformer and ViT with satellite imagery for urban-indicator prediction and text–image matching [Yan et al., 2024].Its listed training approach is next-token prediction using location descriptions.Source: A Summary Tables of Scientific LLMs · p. 35
• FourCastNet, Pangu-Weather, ClimaX, FengWu, W-MAE, and FuXi apply vision-transformer or related architectures to climate forecasting [Pathak et al., 2022; Nguyen et al., 2023c; Chen et al., 2023b; Man et al., 2023; Chen et al., 2023c].Their listed approaches include regression on ERA5 or CMIP6 and masked image modeling on ERA5; ClimaX also covers climate projection and downscaling.Source: A Summary Tables of Scientific LLMs · p. 35