PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers

Ming Hu, Chenglong Ma, Wei Li, Wanghan Xu, Jiamin Wu, Jucheng Hu, Tianbin Li, Guohang Zhuang, Jiaqi Liu, Yingzhou Lu, Ying Chen, Chaoyang Zhang, Cheng Tan, Jie Ying, Guocheng Wu, Shujian Gao, Pengcheng Chen, Jiashi Lin, Haitao Wu, Lulu Chen, Fengxiang Wang, Yuanyuan Zhang, Xiangyu Zhao, Feilong Tang, Encheng Su, Junzhi Ning, Xinyao Liu, Ye Du, Changkai Ji, Pengfei Jiang, Cheng Tang, Ziyan Huang, Jiyao Liu, Jiaqi Wei, Yuejin Yang, Xiang Zhang, Guangshuai Wang, Yue Yang, Huihui Xu, Ziyang Chen, Yizhou Wang, Chen Tang, Jianyu Wu, Yuchen Ren, Siyuan Yan, Zhonghua Wang, Zhongxing Xu, Shiyan Su, Shangquan Sun, Runkai Zhao, Zhisheng Zhang, Dingkang Yang, Jinjie Wei, Jiaqi Wang, Jiahao Xu, Jiangtao Yan, Wenhao Tang, Hongze Zhu, Yu Liu, Fudi Wang, Yiqing Shen, Yuanfeng Ji, Yanzhou Su, Tong Xie, Hongming Shan, Chun-Mei Feng, Zhi Hou, Diping Song, Lihao Liu, Yanyan Huang, Lequan Yu, Bin Fu, Shujun Wang, Xiaomeng Li, Xiaowei Hu, Yun Gu, Ben Fei, Benyou Wang, Yuewen Cao, Minjie Shen, Jie Xu, Haodong Duan, Fang Yan, Hongxia Hao, Jielan Li, Jiajun Du, Yanbo Wang, Imran Razzak, Zhongying Deng, Chi Zhang, Lijun Wu, Conghui He, Zhaohui Lu, Jinhai Huang, Wenqi Shao, Yihao Liu, Siqi Luo, Yi Xin, Xiaohong Liu, Fenghua Ling, Yuqiang Li, Aoran Wang, Siqi Sun, Qihao Zheng, Nanqing Dong, Tianfan Fu, Dongzhan Zhou, Yan Lu, Wenlong Zhang, Jin Ye, Jianfei Cai, Yirong Chen, Wanli Ouyang, Yu Qiao, Zongyuan Ge, Shixiang Tang, Junjun He, Chunfeng Song, Lei Bai, Bowen Zhou

arXiv:2508.21148v2cs.CLcs.AI

TL;DR

Scientific datasets remain constrained in scope, granularity, and modality coverage. This survey synthesizes Sci-LLM models, data, and evaluation, identifying distinctive data challenges and persistent issues in data quality, representation, and knowledge updating.

Abstract

from arXiv · show

Scientific Large Language Models (Sci-LLMs) are transforming how knowledge is represented, integrated, and applied in scientific research, yet their progress is shaped by the complex nature of scientific data. This survey presents a comprehensive, data-centric synthesis that reframes the development of Sci-LLMs as a co-evolution between models and their underlying data substrate. We formulate a unified taxonomy of scientific data and a hierarchical model of scientific knowledge, emphasizing the multimodal, cross-scale, and domain-specific challenges that differentiate scientific corpora from general natural language processing datasets. We systematically review recent Sci-LLMs, from general-purpose foundations to specialized models across diverse scientific disciplines, alongside an extensive analysis of over 270 pre-/post-training datasets, showing why Sci-LLMs pose distinct demands -- heterogeneous, multi-scale, uncertainty-laden corpora that require representations preserving domain invariance and enabling cross-modal reasoning. On evaluation, we examine over 190 benchmark datasets and trace a shift from static exams toward process- and discovery-oriented assessments with advanced evaluation protocols. These data-centric analyses highlight persistent issues in scientific data development and discuss emerging solutions involving semi-automated annotation pipelines and expert validation. Finally, we outline a paradigm shift toward closed-loop systems where autonomous agents based on Sci-LLMs actively experiment, validate, and contribute to a living, evolving knowledge base. Collectively, this work provides a roadmap for building trustworthy, continually evolving artificial intelligence (AI) systems that function as a true partner in accelerating scientific discovery.

I. INTRODUCTION

The introduction frames Sci-LLMs as co-developing with heterogeneous scientific data, including visual and symbolic knowledge beyond ordinary text, and surveys the specialized representations and evaluation this requires. It also situates these modalities in cross-disciplinary synthesis and scientific reasoning, noting challenges in visual-text alignment and representation robustness while connecting data foundations to agentic discovery.

I. INTRODUCTION

  • Sci-LLMs progressed through transfer learning, scaling, instruction following, and agentic science, with agents planning and iterating across discovery workflows.Intern-S1 was continually pre-trained on 2.5 trillion tokens and surpassed existing closed-source state-of-the-art models on molecular synthesis, reaction condition prediction, and crystalline thermodynamic stability prediction; agent systems also integrate tools and knowledge-graph orchestration.Source: §I · pp. 6–7
  • Scientific corpora span heterogeneous modalities and scales, hierarchical knowledge, domain-specific representations, and uncertainty that text-based approaches do not adequately preserve.The introduction calls for architectures that preserve domain-specific invariance, enable cross-modal reasoning, capture multi-resolution dependencies, and propagate error bounds.Source: §I · pp. 7–8
  • Static science benchmarks and final-answer tests miss discovery skills; newer assessments target hypothesis generation, scientific workflows, collaboration, and scientifically constrained outputs.Examples include ResearchBench for discovery across 12 disciplines and process-oriented evaluations such as ScienceAgentBench.Source: §I · p. 8
  • The survey links scientific data foundations to agent frontiers through a unified knowledge framework, cross-domain Sci-LLM review, analyses of over 270 training and over 190 evaluation datasets, and a closed-loop data agenda.It aims to provide a roadmap for trustworthy, continually evolving Sci-LLMs that support scientific discovery.Source: §I · p. 9

A. Taxonomy of Scientific Data

  • Scientific data span textual, visual, symbolic, structured, time-series, and multi-omics formats, motivating multimodal models that integrate heterogeneous data.The taxonomy treats multi-omics as a special case because it combines biological layers such as genomics, transcriptomics, and proteomics.Source: A. Taxonomy of Scientific Data · p. 9

1) Textual Formats:

  • Scientific text ranges from raw notebooks and domain repositories through scholarly and educational resources to regulatory records, observational streams, and computational artifacts.These materials support reproducibility, knowledge synthesis, and AI-driven discovery.Source: 1) Textual Formats · pp. 9–10

2) Visual Data:

  • Scientific visual data include instrumental images and diagrams spanning microscopy, clinical imaging, Earth observation, and astronomy across scales and modalities.Examples include microscopy and spectroscopy in physics and materials science, cellular and radiological imaging in life sciences, and satellite and astronomical observations.Source: 2) Visual Data · pp. 10–12
  • Scientific visual data span microscopy, molecular diagrams, schematics, and climate or cosmic visualizations; MaCBench and MMSci [75] support cross-disciplinary synthesis, while semantic alignment remains difficult.Dense visual information is particularly challenging to align with text for complex phenomena in molecular biology, materials science, and mathematical physics.Source: 2) Visual Data · p. 12

3) Symbolic Representations:

  • Scientific symbolic data encode entities, relationships, and laws in compact, interpretable, machine-processable forms that support automated reasoning, transformation, and discovery.Examples include molecular string notations, crystallographic files, and parameterized equations for physics and Earth system modeling.Source: 3) Symbolic Representations · p. 12
  • SMILES is a prevalent molecular string encoding but is syntactically fragile; SELFIES guarantees 100% validity, yet SMILES remains more expressive for molecular optimization.SMILES describes structures with ASCII strings, while SELFIES uses grammar rules and localized tokens for robust parsing.Source: 3) Symbolic Representations · p. 12
  • CIF is the standard structured format for crystalline materials, encoding unit-cell parameters, atomic positions, symmetries, and experimental metadata.CIF representations underpin databases including ICSD and the Materials Project, and benchmarks such as MatBench.Source: 3) Symbolic Representations · p. 12
  • Symbolic methods support scientific discovery: symbolic derivation improves physics problem-solving accuracy over numeric-first approaches [68], and equation graphs can recover physical laws [220].LLM-powered Dual Reasoning Symbolic Regression combines language-model reasoning with reflective optimization for equation extraction [69]; PhyE2E generates dimensionally consistent formulas from astronomy data [221].Source: 3) Symbolic Representations · p. 12
  • Earth science uses symbolic formula fitting and regression to model partially understood phenomena, aligning theoretical predictions with observations through parameterization and regression methods.Examples include Navier-Stokes equations, atmospheric motion, seismic wave equations, and shallow-water equations; methods include least squares and Bayesian inference.Source: 3) Symbolic Representations · pp. 12–13

4) Structured Data:

Scientific data ranges from structured tables and temporal records to integrated multi-omics, while scientific knowledge is organized across levels from observations to theory. The framework emphasizes the distinct representations and interactions needed to connect these data and knowledge forms.

4) Structured Data:

  • Structured scientific data progresses from tables and relational databases to ontologies and knowledge graphs that encode semantics and relationships for interoperability and reasoning.Repositories also use standardized exchange formats and persistent identifiers to support traceability and cross-disciplinary use.Source: 4) Structured Data · p. 13
  • Molecular data can be encoded as graphs, SMILES or SELFIES strings, node identities, and adjacency matrices, whose edge weights denote bond types.Hydrogens are typically omitted from SMILES and SELFIES strings.Source: Figure 10 · p. 13

5) Time-Series Data:

  • Scientific time series capture phenomena across temporal scales, from molecular dynamics and astronomical brightness observations to EEG, meteorology, and seismic waveforms.Standardizing diverse time-series formats supports cross-disciplinary AI applications.Source: 4) Structured Data · p. 14, 5) Time-Series Data · p. 14

6) Multi-omics Integration:

  • Multi-omics integrates seven data layers, from genomics and epigenomics to microbiome and exposome, offering complementary views of biological processes.DNA, RNA, and proteins have both symbolic sequence representations and three-dimensional structures.Source: 6) Multi-omics Integration · pp. 15–16

B. Hierarchical Structure of Scientific Knowledge

  • The proposed hierarchy has five levels—factual, theoretical, methodological-technological, modeling-simulation, and insight—and links them through iterative feedback.Its structure is presented as a basis for organizing knowledge and generating scientific insights.Source: B. Hierarchical Structure of… · p. 16, Figure 14 · p. 17

1) Factual Level:

  • The factual level consists of direct observations and measurements, whose scale and modality vary widely and whose interpretation must account for uncertainty and noise.Examples span high-volume particle-physics data, high-frequency strain signals, and multi-channel neural recordings.Source: 1) Factual Level · pp. 16–17

2) Theoretical Level:

  • The theoretical level includes mathematical formalisms, conceptual models, classifications, networks, computational models, and statistical frameworks.These domain-specific representations transform raw data into scientific knowledge that supports discovery and prediction.Source: 2) Theoretical Level · p. 17

3) Methodological and Technological Level:

Scientific knowledge develops through methods, models, and insights that interact across levels, while Sci-LLMs face challenges in interpreting and integrating scientific information. The section outlines implications for model capabilities, dataset quality, and evaluation.

3) Methodological and Technological Level:

  • Methods and instruments connect raw observations with theory, enabling testable hypotheses and extending what scientists can observe and investigate.Examples include CRISPR-Cas9, high-throughput sequencing, computational methods, and cryo-electron microscopy.Source: 3) Methodological and Technological… · pp. 17–18

4) Modeling and Simulation Level:

  • Numerical simulations let researchers test hypotheses and predict difficult or costly phenomena, while multi-scale models connect processes across different scales.Verification and validation compare computational accuracy and predictive validity against experimental data.Source: 4) Modeling and Simulation Level · p. 18

5) Insight Level:

  • Scientific insight can arise from cross-disciplinary synthesis and attention to anomalies, while human judgment remains important for distinguishing correlation from causation.The section describes machine learning as useful for finding patterns across omics data, with human-AI collaboration supporting interpretation.Source: 5) Insight Level · p. 18

6) Dynamic Interactions and Evolution:

  • Scientific progress involves bottom-up induction, top-down deduction, and horizontal method transfer, with knowledge frameworks evolving as new theories incorporate earlier ones.The section describes gravitational-wave detection as confirmation of a theoretical prediction and an example of theory guiding experimental development.Source: 6) Dynamic Interactions and Evolution · pp. 18–19

7) Implications for Sci-LLMs:

  • Sci-LLMs are framed as needing capabilities across all five knowledge levels, from parsing heterogeneous observations to cross-domain synthesis and hypothesis generation.At the modeling level, they can interface between natural-language questions and computational engines.Source: 7) Implications for Sci-LLMs · p. 19

C. Key Challenges in Scientific AI

D. Quality Standards for Scientific Datasets

E. Dimensions for Evaluating Scientific AI

III. SCIENTIFIC LARGE LANGUAGE MODELS

This section surveys scientific LLMs, from general-model architectures and training to broad-purpose and domain-specific systems across life sciences, including medicine, agriculture, neuroscience, and astronomy. It describes how these models adapt to scientific modalities, tasks, and workflows through domain-specific data, modality integration, and adaptation.

A. Introduction of Large Language Models

  • LLMs typically use decoder-only transformers, convert text and encoded modalities into tokens, and rely on large-scale next-token pretraining followed by post-training to activate task-specific capabilities.Domain-specific encoders can make scientific inputs such as images and protein sequences compatible with LLM backbones.Source: A. Introduction of Large Language Models · p. 22

B. General-purpose Sci-LLMs

  • General-purpose Sci-LLMs progress from limited fine-tuning toward large scientific pretraining and test-time reasoning, with Intern-S1 spanning modalities and outperforming other systems across scientific benchmarks.Galactica was trained on 106 B scientific tokens; reasoning-oriented approaches include DeepSeek-R1 and Qwen3, while Kimi K2 uses agentic data and Gemini 2.5 Pro handles multimodal inputs with a 1-million-token context. Intern-S1 was trained on over 5 trillion tokens, including 2.5T scientific tokens.Source: B. General-purpose Sci-LLMs · pp. 22–24

1) Physics:

  • Physics Sci-LLMs range from code-driven simulation and PDE operator learning to domain-specific retrieval, with POSEIDON matching FNO performance using 20 samples versus 1024.LLMPhy iteratively refines physical parameters using simulation feedback, while Xiwu outperforms Vicuna-13B in 95% of win-or-draw rate and surpasses GPT-4 on most BOSS code-generation tasks.Source: 1) Physics · pp. 24–25

2) Chemistry:

  • Chemistry models span text, molecular structures, and images, with Chem3DLLM encoding 3D structures for language-model integration and using chemical-validity rewards to promote physical plausibility.The surveyed applications include molecular design, reaction prediction, retrosynthesis, catalyst discovery, quantum chemistry, and materials science.Source: 2) Chemistry · p. 25

3) Materials Science:

  • About 91.7% accuracy on a held-out QA benchmark is reported for Qwen2-KG, which combines chain-of-thought retrieval with materials knowledge-graph facts.Other surveyed approaches include sequence-based property prediction, crystal generation, and synthesis planning; CrystaLLM generates metastable materials with about 49% frequency, reaction-prediction accuracy reaches about 90%, and CSLLM reports approximately 98.6% accuracy on synthesizability prediction. MechGPT supports retrieval, hypothesis generation, and interpretable ontological knowledge graphs.Source: 3) Materials Science · pp. 25–26

4) Life Sciences:

  • Multi-omics models range from sequence-trained genomic and protein encoders to LLM systems for controllable generation, cross-modal alignment, and dialogue-based analysis.EVO2 scales to 9.3 trillion DNA bases and up to 1M-token context, while ESM3 unifies sequence, structure, and function; LLaMA-Gene and Evolla exemplify instruction-following integration and protein-centric question answering.Source: 3) Materials Science · p. 26, 4) Life Sciences · p. 26
  • Molecular and cellular models support property prediction and single-cell analysis, while LLM-based drug-design approaches reduce synthesis trials by 50-70% in virtual screening benchmarks.scGPT is pretrained on over 33 million normal human cells and supports cell-type annotation, batch correction, perturbation-response prediction, and gene-network inference.Source: 4) Life Sciences · p. 27
  • Medical LLMs commonly begin with supervised fine-tuning on medical instructions, while further biomedical pretraining and safety-oriented training address limits of fine-tuning alone.Examples include multilingual Apollo for hospital deployment and models such as PMC-LLaMA, HuatuoGPT-II, and Me-LLaMA that scale biomedical pretraining; CHIMED-GPT also uses RLHF to enhance response safety.Source: 4) Life Sciences · p. 27
  • Medical MLLMs target report generation and VQA, with LLaVA-Med [537] adapting LLaVA and CXR-LLaVA [538] and Radiology-LLaMA2 [539] generating clinical-style chest X-ray reports.Med-Flamingo [540] is continually pre-trained on paired and interleaved medical image-text data and supports few-shot VQA without further VQA-dataset fine-tuning.Source: 4) Life Sciences · pp. 27–28
  • Medical systems broaden task coverage with large image-caption and instruction datasets, long-context processing, and methods for complex reasoning and image grounding.HuatuoGPT-Vision [541] and GMAI-VL [542] target diverse departmental tasks; MedGemma [543] handles long-context videos and electronic health records. HuatuoGPT-o1 [544], Medground-r1 [545], and GMAI-VL-R1 [546] use verifier-based rewards, spatial-semantic rewards, or reinforcement learning on verifiable VQA data, respectively.Source: 4) Life Sciences · p. 28
  • AgroGPT [552] outperforms generalist models and human baselines in image-based agronomic reasoning on AgroEvals.Its vision-language variants are fine-tuned on AgroInstruct, a dataset of 70k synthetic QA pairs created from agricultural images.Source: 4) Life Sciences · p. 28
  • Neuroscience models span literature-based question answering, EEG classification, neural-token modeling, and brain-signal decoding into language, using multimodal alignment and subject-agnostic approaches.BrainGPT outperformed general models and human experts on BrainBench; EEG-GPT [554] reports strong few-shot performance with 2% of training data and tree-of-thought reasoning with EEG tools. NeuroLM [555] converts EEG into discrete tokens for multi-task decoding; UMBRAE, MindGPT, and MindLLM [559] align brain activity with visual or language representations. UniMind [560] supports diverse EEG tasks without task-specific fine-tuning, while Neuro-GPT uses self-supervised masked EEG reconstruction.Source: 4) Life Sciences · pp. 28–29
  • Astronomy-focused models use continual pre-training on scientific literature followed by task-specific fine-tuning, often with parameter-efficient methods such as LoRA.AstroLLaMA [562], AstroLLaVA [563], and AstroSage build on LLaMA-family architectures; multimodal models add visual encoders and projection layers to integrate astronomical images with text.Source: 5) Astronomy · p. 29

5) Astronomy:

The survey connects scientific models’ progress to specialized data and domain adaptation, spanning astronomy and Earth science through cross-scale pretraining and emerging data–task approaches.

5) Astronomy:

  • AstroLLaMA reduced perplexity 32.5% versus LLaMA-2, AstroLLaVA slightly outperformed LLaVA 1.5 on galaxy descriptions, and AstroSage-LLaMA-3.1 reached 86.2% accuracy.AstroSage-LLaMA-3.1’s accuracy was measured without inference mode on AstroMLab-1; the benchmark questions were not used for training.Source: 5) Astronomy · p. 29

6) Earth Science:

  • Earth-science systems pair specialized instruction datasets with models for fine-grained remote-sensing interpretation, high-resolution imagery, and multimodal or temporal Earth observation.Examples include GeoPixel’s pixel-level grounding and EarthDial’s instruction data spanning RGB, SAR, and multispectral inputs.Source: 6) Earth Science · pp. 29–30

D. Sci-LLMs Analysis

  • Roughly three quarters of surveyed models are text-only and one quarter multimodal; the survey anticipates growing demand for models that synthesize heterogeneous scientific signals.The text attributes the current imbalance to text-source dominance and the scarcity and cost of fine-grained multimodal supervision.Source: D. Sci-LLMs Analysis · p. 30
  • Open-source LLaMA and Qwen families lead the base-model landscape, while 7B models are most common and models of 70B or more remain comparatively rare.The survey relates this size distribution to deployment constraints and notes that small-to-mid-sized models can be competitive on narrow scientific tasks when paired with retrieval or fine-tuning.Source: D. Sci-LLMs Analysis · pp. 30–31
  • The survey identifies provenance-grounded generation, stronger cross-modal alignment, tool-connected agents, and temporal adaptation as recurring data–task interface themes.It describes citations and retrieval logs as trust scaffolds and tools such as simulations and code execution as supports for reproducible scientific work.Source: D. Sci-LLMs Analysis · p. 31

IV. SCIENTIFIC DATA FOR PRE-TRAINING

  • Scientific pretraining must capture diverse knowledge and modalities; Intern-S1 devotes 2.5T tokens, or 45.8% of its corpus, to scientific content across six domains.The survey contrasts this specialization with general pretraining mixtures that include webpages, code, papers, and Q&A.Source: §IV · p. 31

A. Physics, Chemistry and Material Sciences: the Foundation for Understanding the Material World

  • Physical-science pretraining draws on simulations and molecular or reaction corpora, while fragmented, inconsistently formatted observational data constrain physics data use.The survey identifies integrating simulation outputs, experimental measurements, and literature knowledge as a path toward reasoning across atomic-scale and engineered-material systems.Source: 1) Physics · pp. 32–33, 3) Materials Science · p. 33

1) Molecular and Cell Biology:

  • Life-science pretraining must represent biological knowledge across molecular, cellular, tissue, organismal, clinical, and ecological scales using heterogeneous data.Existing resources include sequence-focused datasets, such as UniRef50 and UniRef90, which contain approximately 49 million protein sequences each after clustering at their respective identity thresholds.Source: 1) Molecular and Cell Biology · p. 33

2) Multi-Omics:

Scientific training corpora span diverse data types and disciplines, but coverage remains uneven and data can be costly or difficult to assemble. The survey describes domain-specific resources and gaps across pre-training and post-training.

2) Multi-Omics:

  • NatureLM combines 35 biomedical corpora across sequences, clinical narratives, literature, and imaging captions, while LLaMA-Gene aligns natural-language queries with molecular knowledge graphs.ChatNT combines DNA and English tokens across tasks involving DNA, RNA, and protein processes.Source: 2) Multi-Omics · p. 34

3) Neuroscience:

  • Neuroscience pre-training draws on literature and modality-specific encoders trained on signals including EEG, fMRI, and MEG.LaBraM integrates over 2534.78 hours of EEG data from about 20 public and proprietary datasets.Source: 3) Neuroscience · p. 34

4) Healthcare and Medical Science:

  • Healthcare datasets span clinical text and image-text pairs, but longitudinal EHR data are scarce and clinical reports face ethical-review and privacy constraints.Medical image data, particularly radiology, is prominent; multimodal annotation quality varies from noisy captions to partially validated annotations.Source: 4) Healthcare and Medical Science · pp. 34–35

5) Agriculture:

  • Agricultural pre-training uses multilingual agronomy literature, textbooks, and genomic databases, but domain-specific datasets have not been publicly released.Their non-release hinders reproducibility and further research.Source: 5) Agriculture · p. 35

C. Astronomy and Earth Science: Understanding Our Planet

  • Astronomical pre-training remains fragmented across modalities, with comprehensive datasets integrating images, spectra, and time series largely absent.Earth science data are dispersed and heterogeneous; existing pre-training efforts remain text-centric and modest in scale.Source: 1) Astronomy · p. 35, 2) Earth Science · pp. 35–36

D. Pre-training Data Analysis

  • Academic papers, SMILES strings, and radiology images dominate pre-training corpora, while modalities and task types remain unevenly covered.The survey also identifies raw text and classification as heavily represented task types.Source: D. Pre-training Data Analysis · p. 36
  • Pre-training gaps include weakly grounded labels, inconsistent standards, scarce open corpora, governance constraints, and unresolved scale–quality trade-offs.Idealized physics simulations may transfer poorly to instrument-specific observations; clinical data can be smaller or temporally stale.Source: D. Pre-training Data Analysis · p. 36

V. SCIENTIFIC DATA FOR POST-TRAINING

  • Scientific post-training datasets are shifting toward larger, multimodal, reasoning-rich collections, but science content is only 2.9% of Cambrian-7M.Physics resources support short-form problem solving, yet most remain text-only and rarely represent diagrams or other real-world multimodal tasks; chemistry post-training data is limited by experimental costs.Source: §V · pp. 36–37, 1) Physics · p. 37, 2) Chemistry · p. 37

3) Materials Science:

Scientific post-training datasets are broadening across modalities and disciplines, while reasoning traces and workflow-linked examples remain uneven or scarce. The survey also reviews specialized benchmarks spanning scientific knowledge and reasoning.

3) Materials Science:

  • Materials science datasets remain scarce and often repurposed from pre-training corpora, with limited visual data and few multistep reasoning or integrated workflow examples.The described gaps include workflows combining molecular design with property calculation and analysis.Source: 3) Materials Science · p. 37

4) Life Sciences:

  • Life-sciences post-training spans sequences, molecular graphs, clinical images, and neural recordings, requiring models to work across biological scales.The surveyed datasets cover molecular and cell biology, multi-omics, medicine, and neuroscience.Source: 4) Life Sciences · pp. 37–38

5) Astronomy:

  • Astronomy post-training has expanded from text corpora to image-caption pairs and synthetic question-answer sets for domain-specific tasks.Astro-NER illustrates that fine-grained named-entity recognition still uses manual curation.Source: 5) Astronomy · p. 38

6) Earth Science:

  • Earth-science post-training datasets include GIS-derived visual QA, weather reasoning, sonar QA, and multimodal resources across optical, radar, and infrared imagery.Automated annotation is also described as enabling multistep reasoning over complex Earth-system interactions.Source: 6) Earth Science · pp. 38–39

B. Post-training Data Analysis

  • Post-training corpora commonly convert structured domain knowledge into instruction-response pairs, while multimodal datasets bridge visual and textual reasoning across fields.Scalable synthesis mines literature, simulations, and databases for instruction-response pairs and reasoning traces; multi-agent validation is one cited approach to maintaining fidelity.Source: B. Post-training Data Analysis · p. 39
  • Across domains, post-training is shifting toward multimodal and multi-domain corpora, but source distributions show domain-specific biases and cross-domain imbalances.Reasoning supervision beyond simple QA is growing, though large-scale public chain-of-thought corpora remain scarce outside biomedical sciences.Source: B. Post-training Data Analysis · p. 39
  • Real workflow-linked reasoning traces remain scarce, some chain-of-thought data lack extensive expert validation, and licensing, privacy, and standardization hinder reuse.The survey also notes uneven multimodal coverage, including a shortage of large-scale physics datasets pairing problems with diagrams or simulations.Source: B. Post-training Data Analysis · p. 40

VI. EVALUATION OF SCI-LLMS

  • Scientific benchmarks span educational, research-oriented, multimodal, and agent tasks, assessing capabilities from domain knowledge and reasoning to clinical workflows.Examples include equation-discovery benchmarks in physics, multimodal chemistry and materials assessments, and medical-agent evaluations in virtual clinical environments.Source: 1) Physics · p. 40, 2) Chemistry · p. 40, 3) Materials Science · p. 41

4) Life Sciences:

Scientific evaluation is shifting from static question-answering toward domain-shaped, process-aware assessment, but current benchmarks and scoring practices remain uneven. Across domains, the survey highlights benchmark coverage and annotation limitations, and discusses agent judging and test-time learning as emerging approaches.

4) Life Sciences; 6) Earth Science; 7) General Science:

  • Life-science evaluation is moving toward interactive, end-to-end decision-making, while broad scientific benchmarks span exam-style QA, multimodal figure/image QA, and symbolic or programmatic reasoning.Across domains, foundation models still have gaps in specialized representations and cross-modal reasoning.Source: 4) Life Sciences · p. 41, 7) General Science · p. 42, 6) Earth Science · p. 42

7) General Science:

  • 80–95% accuracy on MMLU-Pro contrasts with 2–10% on HLE and 20–40% on SFE in most scientific domains, exposing a substantial gap on frontier scientific tests.Chemistry performs best on HLE, but results remain poor.Source: 7) General Science · p. 42

B. Evaluation Data Analysis

  • Most scientific domains rely on a dominant evaluation-source type, and current benchmark corpora emphasize text-based QA over hypothesis generation, experimental design, and cross-domain reasoning.The source distributions are normalized within each domain.Source: B. Evaluation Data Analysis · p. 43, Figure 25 · p. 43

1) Tiered Regime in Data Generation and Annotation:

  • Benchmark annotation uses expert curation, human-in-the-loop pipelines, and automated systems, trading quality, scalability, and resources across tiers.Semi-automated examples pair LLMs or tools with expert review, while automated LLM annotation risks circularity, contamination, and error propagation.Source: 1) Tiered Regime in Data Generation… · p. 43

2) Skewed Knowledge Level with Increasing Difficulty:

  • Difficulty labels are often absent, and labeled datasets skew toward expert or undergraduate levels, with few intermediate slices and an increasing polarization in 2024–2025.The expert-level sets demand domain knowledge and multimodal or cross-domain reasoning, not only factual recall.Source: 2) Skewed Knowledge Level with… · pp. 43–44

3) Shift towards Domain-Specific Metrics:

  • Evaluation is moving from accuracy toward domain-specific metrics, which can better align scores with scientific targets but remain vulnerable to metric gaming and inconsistent cross-dataset portfolios.The survey proposes calibration and uncertainty reporting, process-aware scoring, factuality checks, and standardized multi-metric dashboards.Source: 3) Shift towards Domain-Specific Metrics · p. 44

C. LLM / Agent as a Judge

  • Agent-as-a-Judge uses dynamic planning and intermediate feedback to assess complex, multi-step discovery processes that a single metric cannot capture.The framework provides intermediate feedback and analysis of the whole process.Source: C. LLM / Agent as a Judge · pp. 44–45

D. Inspiration from Test-Time Learning

  • Test-Time Learning adapts models using unlabeled test samples; reported examples include gains across biomedical tasks and chemistry performance gains that diminish when adaptation runs too long.MedAdapter reports 18.24% improvement for white-box LLMs and 10.96% for black-box LLMs across four biomedical reasoning tasks and eight datasets.Source: D. Inspiration from Test-Time Learning · p. 45

A. Data Collection and Labeling

The section surveys how scientific data are acquired, synthesized, and validated, then examines dataset limitations and systemic quality issues. It also describes a shift toward scientific agents that plan and carry out research tasks.

1) Data Source Heterogeneity and Acquisition Strategies:

  • Scientific sources span academic repositories, web-scraped material, and patents, but publication delays, language imbalance, and variable web-data reliability constrain what they capture.Patent records can provide experimental procedures not often found in academic publications.Source: 1) Data Source Heterogeneity and… · p. 46

2) Annotation Methodologies and Quality Control:

  • Pre-training prioritizes scale and coverage, while post-training adds stricter factual and citation checks; evaluation data receives multilevel questions, answer review, and human validation.Synthesized formats include instruction-response, knowledge and QA pairs, and open-ended evaluation questions.Source: 2) Annotation Methodologies and… · pp. 46–47

3) Cross-Domain Patterns and Domain-Specific Considerations:

  • Scientific datasets can favor accessible positive findings over negative results and idealized conditions, while privacy and proprietary-data constraints narrow demographic, geographic, and agricultural coverage.Medical data require de-identification and access controls, whereas proprietary farming data can limit represented crop varieties and growing conditions.Source: 3) Cross-Domain Patterns and… · p. 47

1) Scarcity of Experimental Data:

  • Experimental data are scarce because acquisition is costly, some phenomena are rare, and heterogeneous formats and missing metadata make integration labor-intensive.Clinical-data access also involves ethical review and privacy constraints.Source: 1) Scarcity of Experimental Data · pp. 47–48

2) Over-reliance on Text Modality Data:

  • Text-heavy corpora often omit raw measurements, negative findings, protocols, and high-quality scientific modalities, limiting quantitative depth and empirical grounding.Text reports may provide averages without the distributions needed to reason about experimental variability.Source: 2) Over-reliance on Text Modality Data · p. 48

3) Representation Gap between Static Knowledge and Dynamic Processes:

  • Static datasets can expire as findings change, leaving models with outdated or contradictory information for evolving phenomena and real-time reasoning.The gap is especially apparent in observational records, experimental results, and QA benchmarks drawn from published sources.Source: 3) Representation Gap between Static… · p. 48

4) Multi-level Biases in Scientific Datasets:

  • Publication, language, domain, author, and institutional biases skew scientific corpora, including an overabundance of positive findings and underrepresentation of some disciplines and voices.Studies with statistically significant findings are reported as up to three times more likely to be published than studies with null results.Source: 4) Multi-level Biases in Scientific… · p. 48

C. Systematic Issues in Data Quality

2) Multi-Agent Collaboration:

The section surveys scientific agent systems, from structured multi-agent collaboration and tool use to autonomous discovery. It argues that agents need data ecosystems that connect scientific evidence, tools, and workflows, while identifying persistent gaps in data and validation.

2) Multi-Agent Collaboration:

  • Capability-aligned roles, structured communication, and formal conflict resolution are the conditions under which multi-agent settings yield their largest gains.MultiAgentBench [84] and communicative multimodal tasks make interaction skills measurable.Source: 2) Multi-Agent Collaboration · p. 50

3) Tool Use:

  • SciToolAgent [59] organizes domain tools in a capability knowledge graph, while Biomni connects 150 tools, 59 databases, and 105 software packages for biomedical analyses.Tool-use systems also employ standardized transport, sandboxes, and reflective monitors to manage orchestration and safety.Source: 3) Tool Use · p. 50

4) Self-evolving Agents:

  • Self-evolving agents update memories, prompts, models, or tool-use policies from experience; scientific examples include STELLA [835], OriGene, and ChemAgent.Reliable self-evolution across long horizons with closed-loop laboratory validation remains rare.Source: 4) Self-evolving Agents · pp. 50–51

5) Evaluation Frameworks and Benchmarking:

  • Scientific evaluation increasingly tests long-horizon workflows with verifiable endpoints, while benchmarks also assess long-context reasoning, simulated discovery, causal discovery, and collaboration.ScienceAgentBench reports that the best agents solved only about a third of its 102 tasks.Source: 5) Evaluation Frameworks and… · p. 51

6) Autonomous Scientific Discovery:

  • Autonomous discovery systems span chemistry, life sciences, and materials design, but must balance valid proposals with novelty and integrate experimental validation amid real-world uncertainty.InternAgent [852] demonstrates integrated pipelines from idea generation to experimental validation.Source: 6) Autonomous Scientific Discovery · p. 51

B. Data Ecosystems for Sci-LLMs

2) Building an Operating System-level Interaction Protocol:

  • An operating-system-level protocol would let agents decompose objectives, invoke scientific tools, and incorporate tool outputs into their reasoning; the proposed architecture emphasizes actionable data, continuous updates, and traceability.These principles link experimental context and data provenance with cross-modal knowledge integration.Source: 2) Building an Operating System-level… · p. 52, 3) Design Principles for… · p. 53

4) Sustainable Data Sharing Mechanism:

4) LLM Knowledge Update and Version Control:

Scientific knowledge changes rapidly, while static training snapshots and retrieval-augmented approaches may not keep models current or reliably resolve source relevance and conflicts. The section calls for integrated, continuously updated scientific data ecosystems with provenance and community incentives.

4) LLM Knowledge Update and Version Control:

  • Static literature snapshots can quickly become outdated, and retrieval-augmented approaches may fall short on relevance filtering, source attribution, and conflicting information.The section calls for automated updates from peer-reviewed publications, preprints, and curated datasets with version control and traceability.Source: 4) LLM Knowledge Update and Version… · p. 54

1) Integrated Scientific Data Ecosystems:

  • Integrated scientific data ecosystems would connect apparatus, simulations, theory, and published knowledge into active streams that propagate new experimental results while tracking provenance.The section also calls for incentives that recognize data curation and sharing as research contributions.Source: 1) Integrated Scientific Data Ecosystems · p. 55

2) Automated Scientific Data Standardization Pipeline:

The section calls for automated data standardization pipelines to make Sci-LLM development scalable, transparent, and reproducible. It emphasizes cleaning, validation, enrichment, and versioned preparation of heterogeneous scientific data.

2) Automated Scientific Data Standardization Pipeline:

  • Automated standardization pipelines should replace manual, ad hoc curation with scalable preparation of high-quality, accessible scientific data.The stated goal is standardized data that the scientific community can readily access.Source: 2) Automated Scientific Data… · p. 55
  • Pipelines should automatically clean, validate, and enrich raw scientific data across heterogeneous forms and modalities to provide high-fidelity model inputs.Source: 2) Automated Scientific Data… · p. 55
  • Robust data versioning and reproducible preparation workflows are needed to make Sci-LLM development transparent and reproducible.Source: 2) Automated Scientific Data… · p. 55

3) Comprehensive Evaluation System:

The section argues that comprehensive Sci-LLM evaluation must address both models and data. It calls for assessments of scientific reasoning and creativity alongside systematic measures of dataset quality, while also identifying architectural innovations as a future direction.

3) Comprehensive Evaluation System:

  • Model evaluation should use standardized domain-specific benchmarks for reasoning depth, factual accuracy, and scientific creativity across multimodal, multistep tasks.Source: 3) Comprehensive Evaluation System · p. 55
  • Data evaluation needs a systematic framework covering AI-readiness, completeness, relevance, timeliness, usability, and accessibility.The section notes that current approaches often fail to capture how data supports model capabilities.Source: 3) Comprehensive Evaluation System · p. 55
  • Future architectures should embed physical laws, causal structures, and domain-specific constraints into model design.Source: 4) Advanced Scientific Reasoning · p. 55

4) Advanced Scientific Reasoning:

Advanced scientific reasoning calls for systems that combine symbolic manipulation, generalize compositionally, and represent uncertainty. Scientific agents should also propose hypotheses, design experiments, and iterate on empirical results, supported by safe tools and collaboration protocols.

4) Advanced Scientific Reasoning:

  • Scientific reasoning should combine symbolic manipulation of equations and chemical structures with compositional generalization and explicit uncertainty propagation.The passage frames neural-symbolic integration as essential where interpretability and correctness are paramount.Source: 4) Advanced Scientific Reasoning · p. 55
  • Scientific agents should propose testable hypotheses, design experiments to resolve uncertainties, and iterate based on empirical results.Source: 4) Advanced Scientific Reasoning · p. 55
  • These agents require safe laboratory and simulation protocols, standardized interfaces for tools and databases, and frameworks for multi-agent collaboration.The envisioned systems could autonomously explore hypothesis spaces too vast for human investigation.Source: 4) Advanced Scientific Reasoning · p. 55

6) From Sci-LLMs to Scientific Discovery:

Sci-LLMs are envisioned as collaborators that move beyond literature review toward hypothesis generation and scientific discovery, while human creativity and ethical oversight remain important.

6) From Sci-LLMs to Scientific Discovery:

5) Autonomous Scientific Agents:

Responsible scientific AI development requires ethical governance, equitable access, and careful consideration of accountability, validation, and environmental impact.

5) Autonomous Scientific Agents:

  • Robust ethical frameworks and governance structures are necessary for increasingly capable scientific AI systems.Source: 5) Autonomous Scientific Agents · p. 55
  • Equitable access matters as scientific AI begins influencing research directions and resource allocation, to prevent further concentration of scientific capabilities.Source: 5) Autonomous Scientific Agents · p. 55
  • Attribution, accountability, and validation for AI-generated discoveries require careful consideration and community consensus.Source: 5) Autonomous Scientific Agents · p. 55
  • The environmental impact of large-scale model training must be balanced against potential contributions to sustainability science.The passage calls for innovations in efficient training and model architectures.Source: 5) Autonomous Scientific Agents · p. 55

Related papers

Loading 2508.21148v2…