PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM4SR: A Survey on Large Language Models for Scientific Research

Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, Xinya Du

arXiv:2501.04306v1cs.CLcs.DL

TL;DR

Scientific research spans a complex cycle whose productivity is constrained by human creativity, expertise, time, and resources. This survey systematically reviews LLM applications across hypothesis discovery, experimentation, writing, and peer review, synthesizing methodologies, benchmarks, challenges, and future directions. It presents LLMs as advanced productivity tools across modern research while recognizing technical, domain-specific, and ethical constraints.

  • Problem

    Scientific research remains constrained by the creativity, expertise, finite time, and resources available to human researchers.

  • Method

    The survey analyzes LLM applications across four research stages and reviews their methodologies, benchmarks, evaluation methods, limitations, and improvement needs.

  • Results

    The survey provides a holistic analysis of LLM contributions across the scientific workflow, including hypothesis discovery, experiments, writing, and peer reviewing.

  • Takeaways & Limitations

    LLMs represent advanced productivity tools offering new methods across all stages of modern scientific research.

  • Takeaways & Limitations

    The survey focuses on LLMs for scientific research and may omit relevant work by researchers who publish outside computer-science venues.

Abstract

from arXiv · show

In recent years, the rapid advancement of Large Language Models (LLMs) has transformed the landscape of scientific research, offering unprecedented support across various stages of the research cycle. This paper presents the first systematic survey dedicated to exploring how LLMs are revolutionizing the scientific research process. We analyze the unique roles LLMs play across four critical stages of research: hypothesis discovery, experiment planning and implementation, scientific writing, and peer reviewing. Our review comprehensively showcases the task-specific methodologies and evaluation benchmarks. By identifying current challenges and proposing future research directions, this survey not only highlights the transformative potential of LLMs, but also aims to inspire and guide researchers and practitioners in leveraging LLMs to advance scientific inquiry. Resources are available at the following repository: https://github.com/du-nlp-lab/LLM4SR

1 Introduction

The survey examines how LLMs support scientific research across the research cycle, from hypothesis discovery through peer review. It organizes prior work by methodologies, benchmarks, evaluation methods, challenges, and future directions.

  • Scientific research proceeds from background knowledge and hypotheses through experiments, data analysis, manuscript preparation, and peer review, but depends on human creativity, expertise, time, and resources.
  • LLMs extend beyond conventional NLP by processing massive datasets, generating human-like text, and assisting with complex decision-making.
  • The survey covers four applications: hypothesis discovery, experiment planning and implementation, scientific writing, and peer review.
  • For each application, the survey reviews methodologies, benchmarks, evaluation methods, limitations, and areas needing improvement.
  • Compared with narrower prior studies, the survey integrates LLM applications across the entire scientific workflow.
  • Its organization follows sections on hypothesis discovery, experiment planning and implementation, automated paper writing, and LLM-assisted peer review.

2 LLMs for Scientific Hypothesis Discovery

The hypothesis-discovery literature is organized around methodologies, benchmarks, evaluation trends, important progress, and remaining challenges. The survey first situates LLM-based work within literature-based discovery and inductive reasoning.

  • The survey reviews literature-based discovery and inductive reasoning as historical foundations for LLM-based scientific hypothesis discovery.
  • It then synthesizes methods, benchmarks, evaluation development trends, important progress, and main challenges in the discovery task.

2.2 History of Scientific Discovery

Scientific hypothesis discovery developed from literature-based discovery and inductive reasoning. Recent work extends these traditions toward natural-language, web-based, and data-driven discovery across scientific disciplines.

  • LLM-based scientific hypothesis generation emerged from literature-based discovery and inductive reasoning.
  • Literature-based Discovery: Literature-based discovery links concepts through independently created knowledge fragments that are logically related but not jointly retrieved or interpreted.
  • Literature-based Discovery: The ABC model hypothesizes a link between concepts A and C when both co-occur with an intermediate concept B in papers.
  • Literature-based Discovery: Classic literature-based discovery predicts pairwise relations between discrete concepts and does not model the contextual considerations used by human scientists during ideation.
  • Literature-based Discovery: Natural-language extensions constrain generation with context and produce sentences rather than only predicting relations.
  • Inductive Reasoning: Inductive reasoning seeks a broadly applicable rule or hypothesis from specific observations, while scientific discovery applies this reasoning to novel scientific findings.
  • Inductive Reasoning: A valid inductive rule should not conflict with observations, reflect reality, and generalize beyond the observations to cover new information.
  • Inductive Reasoning: Recent approaches apply LLMs to discover social-science hypotheses from news, business reviews, and Wikipedia, and propose data-driven discovery from public and private experimental data.

2.3 Development of Methods

Scientific discovery methods develop by adding structured components for retrieving inspiration, checking hypotheses, ranking candidates, and constructing research questions. The survey traces this trajectory alongside approaches such as literature-based inspiration retrieval, iterative feedback, evolutionary search, and multiple-inspiration composition.

  • Main trajectory: The survey identifies a development trajectory in which scientific discovery methods incorporate increasingly many key components.These include inspiration retrieval, novelty, validity, and clarity checking, evolutionary algorithms, multiple inspirations, hypothesis ranking, and automatic research-question construction.
  • Inspiration retrieval: Literature-based methods retrieve additional knowledge from semantic, knowledge-graph, or citation-graph relationships to inspire hypotheses.SciMON uses semantically similar knowledge, knowledge-graph neighbors, and citation-graph neighbors; SciAgents samples a citation-graph-connected concept, while MOOSE uses LLM-selected candidates.
  • Feedback modules: MOOSE introduced iterative novelty, validity, and clarity feedback as three aspects for refining generated hypotheses.Novelty feedback compares hypotheses with surveys, retrieved papers, or LLM knowledge, while validity feedback generally relies on LLM or neural-model heuristics because experiments are costly.
  • Evolutionary algorithms: Evolutionary approaches treat hypothesis generation as mutation and use selection, recombination, or refinement across candidate hypotheses.FunSearch uses island-based evolution, SGA retains selected offspring and applies crossover, and MOOSE-Chem independently refines and recombines hypotheses.
  • Multiple inspirations: Leveraging multiple inspirations decomposes hypothesis generation into smaller executable steps that progressively address gaps in a preliminary hypothesis.MOOSE-Chem introduced this component for domains such as chemistry and materials science, where multiple inspirations may be needed for a complete publishable hypothesis.

2.4 Benchmarks

Scientific discovery benchmarks are organized around literature-based and data-driven discovery. They represent research questions, background or experimental evidence, and hypotheses while varying in source papers, disciplines, dates, and intended use.

  • Benchmark organization: Automated scientific discovery tasks are divided into literature-based discovery and data-driven discovery, with separate benchmarks designed for each.Literature-based discovery connects knowledge from existing publications, whereas data-driven discovery derives hypotheses from experimental observations.
  • Literature-based discovery: Literature-based benchmarks represent research questions, background surveys, inspiration identification, and hypotheses drawn from scientific papers.Hypotheses may come from abstracts, methodology sections, future-work sections, or limitation sections.
  • Benchmark construction: Publication dates help prevent data contamination when benchmarks evaluate rediscovery of hypotheses with models trained on earlier data.The model’s training data should precede the hypothesis publication date used for rediscovery.
  • Data-driven discovery: Data-driven discovery uses experimental results as observations from which LLMs infer general patterns and formulate hypotheses.The observation space can include public online results and private experimental results available to researchers.
  • Data-driven benchmarks: DiscoveryBench contains 264 manually extracted discovery tasks from over 20 papers and 903 synthetic tasks.Each task supplies a research question and experimental data, requiring a hypothesis supported by those data.
  • Data-driven benchmarks: DiscoveryWorld provides a virtual environment with 120 challenge tasks whose hypotheses reflect real-world patterns.Its virtual setting addresses the cost and domain expertise required by real experiments while encouraging more general discovery methods.

2.5 Evaluation Development Trend

Scientific discovery evaluation uses diverse methods but converges on recurring criteria and evaluator designs. The survey highlights reliability limits of direct or LLM-based assessment and the continuing importance of real experiments.

  • Evaluation criteria: Common evaluation criteria for scientific discovery are novelty, validity, clarity, and significance.Relatedness, interestingness, and helpfulness are used less frequently; validity and feasibility may be interchangeable in many scenarios.
  • Evaluator selection: Evaluators are broadly divided into LLM-based and expert-based approaches.LLM evaluation showed high consistency with expert evaluation in one social-science setting, but chemistry evaluation is challenged by disciplinary complexity and specialized knowledge requirements.
  • Reference dependence: Reference-based evaluation counts key components shared with ground-truth hypotheses as an alternative to less reliable direct evaluation.Direct evaluation assigns scalar scores, whereas reference-based methods compare generated hypotheses against known ground truth.
  • Comparison-based evaluation: Comparison-based evaluation ranks pairs of generated hypotheses and can compare methods without necessarily judging absolute hypothesis quality.This approach is proposed to address limitations of direct scoring by LLM evaluators.
  • Experimental validation: Ultimate evaluation requires real wet-lab experiments, creating challenges for robotics and automatic experiment implementation.This requirement makes experimental verification a practical bottleneck for discovery systems.

2.6 Major Progresses/Achievements on Discovering Hypotheses

Reported achievements show that LLMs can generate scientifically novel and valid hypotheses, with evidence spanning expert evaluations, large-scale comparisons, and rediscovery of recent chemistry and materials findings.

  • Expert-validated generation: LLMs generated novel and valid scientific hypotheses in an initial demonstration confirmed through expert evaluation.Three social-science PhD students evaluated the generated hypotheses for novelty and validity.
  • Human comparison: A large-scale evaluation found that LLMs generated more novel but slightly less valid research hypotheses than human researchers.The conclusion was statistically significant and was based on evaluation by more than 100 NLP researchers.
  • Rediscovery: LLMs trained only on data available through October 2023 rediscovered the main innovations of many 2024 chemistry and materials-science hypotheses.The hypotheses came from papers published in Nature, Science, or similar-level journals and were available online in 2024.

2.7 Challenges and Future Work

Scientific discovery remains constrained by the reliability of hypothesis evaluation, the capabilities of available LLMs, limited understanding of internal reasoning, and small expert-built benchmarks. Future work therefore emphasizes automated experiments, stronger hypothesis generation, richer reasoning structures, and scalable benchmark construction.

  • Challenges and Future Work: Automated experiments are needed because expert evaluation may not reliably validate machine-generated hypotheses, particularly in chemistry.The survey identifies automated experimental execution as the most reliable way to test hypothesis validity.
  • Challenges and Future Work: LLM-based discovery may face an upper performance limit because hypothesis quality depends heavily on the capabilities of existing LLMs.The survey notes that methods using stronger general-purpose LLMs tend to produce higher-quality hypotheses, while augmentation strategies remain unclear.
  • Challenges and Future Work: Current discovery methods rely heavily on high-quality knowledge retrieval, leaving additional internal reasoning structures insufficiently understood.The survey calls for interdisciplinary investigation of reasoning structures in scientific discovery.
  • Challenges and Future Work: Expert-composed discovery benchmarks are usually small, creating an unresolved challenge in scaling accurate and well-structured evaluations.Future work includes automatically collecting accurate and well-structured benchmarks.
  • Challenges and Future Work: The practical bottleneck for automated experimentation differs by discipline, ranging from coding large systems in computer science to laboratory robotics in chemistry and biology.The survey frames experimental execution as a discipline-dependent engineering challenge.

3 LLMs for Experiment Planning and Implementation

LLMs support experiment planning and implementation through task decomposition, iterative refinement, automation, tool integration, and data analysis across scientific domains. Their use is evaluated through varied benchmarks, but hallucinations, prompt sensitivity, slow processing, and weak domain-specific expertise limit reliability.

  • Experimental Planning: LLMs decompose complex experiments into structured subtasks by defining conditions, outputs, execution sequences, and resource dependencies.HuggingGPT parses user queries into task lists, while CRISPR-GPT automates parts of gene-editing experiment planning.
  • Experimental Planning: In-context learning, Chain of Thought, ReAct, reflection, refinement, and simulated expert discussions are used to improve experimental planning.These methods support iterative evaluation, assumption checking, and collaborative refinement of experimental plans.
  • Workflow Automation: LLM-based systems automate repetitive research activities including data preparation, experimental execution, analysis, and reporting.Data preparation can include cleaning, labeling, feature engineering, and synthetic data generation when direct collection is difficult.
  • Workflow Automation: Tool integration, pretraining, fine-tuning, and domain-specific knowledge bases enable LLM agents to acquire capabilities for scientific workflows.ChemCrow uses 18 expert-designed tools for chemical synthesis, while Coscientist integrates LLMs with laboratory automation.
  • Scientific Applications: Antibody generative LLMs can support de novo SARS-CoV-2 antibody design while achieving specificity and diversity and reducing reliance on natural antibodies.This example illustrates LLM use for domain-specific molecular design rather than only workflow orchestration.
  • Benchmarks: Benchmarks evaluate experiment-support systems using task success rate, accuracy, execution consistency, and comparisons with human benchmarks across varied task categories.Table 3 organizes evaluation around experimental design, data preparation, experiment execution and workflow automation, and data analysis and interpretation.
  • Challenges and Future Work: LLMs may produce hallucinated or irrational plans, respond inconsistently to minor prompt changes, process feedback slowly, and struggle to emulate specialized scientific expertise.These limitations affect multi-stage planning, execution reliability, real-time interaction, and cross-domain generalizability.
  • Challenges and Future Work: Future work emphasizes external verification, real-time feedback, adaptive prompt-robust systems, and models tailored to domain-specific experimental requirements.The survey proposes cross-referencing outputs with sound verifiers and dynamically correcting inaccuracies through feedback loops.

4 LLMs for Scientific Paper Writing

LLMs are applied to citation generation, related work synthesis, scientific drafting, and broader paper-production workflows, with evaluation spanning task-specific benchmarks and metrics. Their usefulness is constrained by hallucinations, retrieval and context limitations, and risks to academic integrity and originality.

  • Scope: Scientific writing applications cover citation text generation, related work generation, drafting, and evaluation across these subtasks.The survey organizes this area around methodologies, effectiveness, challenges, metrics, and benchmarks.
  • Citation Text Generation: LLMs generate citation text using manuscript and abstract content, citation-network relationships, and multimodal combinations of structure and textual context.Approaches range from pointer-generator networks to systems such as AutoCite and BACO.
  • Related Work Generation: LLMs assist related work generation by scanning large scientific-article collections and producing initial literature-review drafts using large context windows.Case studies describe ChatGPT-based support for literature review and related work writing.
  • Drafting and Writing: Automated writing systems range from generating scientific definitions and paper drafts to synthesizing complete surveys and supporting hypothesis generation and experiment design.Examples include controllable definition generation, SCICAP, AutoSurvey, AI Scientist, and CycleResearcher.
  • Benchmarks: Citation text generation is evaluated with ALCE across fluency, correctness, and citation quality, while related work generation lacks a universally recognized benchmark.Table 4 summarizes datasets, metrics, and benchmarks for citation, related work, and drafting tasks.
  • Benchmarks: SciGen and SciXGen evaluate reasoning-aware and context-aware scientific text generation using BLEU, METEOR, MoverScore, and human evaluation.SciGen emphasizes reasoning from scientific tables, whereas SciXGen emphasizes integrating external information.
  • Challenges: Writing-oriented LLMs can hallucinate, produce incorrect or irrelevant citations, depend on retrieval systems, and struggle with extensive references because of limited context windows.These constraints can impair factual accuracy, contextual coherence, and comprehensive literature integration.
  • Challenges: Academic use also raises concerns about integrity, authorship, plagiarism, and insufficient originality in machine-generated text.The survey identifies both presenting generated text as one’s own and close imitation of existing literature as risks.

5 LLMs for Peer Reviewing

LLMs are being integrated into peer review through automated review generation and human-assisted workflows, supported by specialized benchmarks and diverse evaluation methods. The survey also identifies technical, infrastructural, ethical, and governance challenges that constrain reliable adoption.

  • 5.1 Overview: LLM-assisted peer review addresses reviewer bias, inconsistent standards, and workload imbalances through automated and human-assisted workflows.Automated systems generate reviews independently, while assisted workflows retain human expertise for academic evaluation.
  • 5.2 Automated Peer Review Generation: Automated review generation uses single-model or multi-model architectures to produce comprehensive assessments of methodology, results, and contributions.Single-model systems simplify implementation but may struggle with complex papers; multi-model systems improve scalability while introducing coordination and consistency challenges.
  • 5.3 Information Extraction and Summarization: Information extraction and summarization systems combine language and vision models to structure visually rich papers and support reviewer comprehension.PaperMage extracts logical structure, figures, and text across modalities, while summarization tools synthesize document content.
  • 5.4 Benchmarks: Systematic comparison remains challenging because automated review generation and assisted workflows use diverse methodologies and require standardized benchmarks.Benchmarks cover comprehensive reviews, specialized assessment tasks, and quality evaluation, with metrics including semantic similarity, coherence, diversity, specificity, and human evaluation.
  • 5.5 Challenges and Future Work: Peer-review LLMs struggle with specialized concepts, methodological judgment, long-document coherence, hallucinations, limited training data, bias, transparency, and feedback homogenization.These limitations can produce inconsistent evaluations and make it difficult to identify inadequate samples, inappropriate statistical tests, or missing controls.
  • 5.5 Challenges and Future Work: Future research prioritizes domain-specific understanding, citation analysis, long-document consistency, human-AI collaboration, governance, and comprehensive evaluation.Proposed systems should integrate with reviewer workflows, track LLM contributions, detect generated content, secure platform integrations, and assess bias across scholarly communities.

6 Conclusion

The survey presents LLMs as advanced productivity tools spanning the scientific research lifecycle, while acknowledging limitations, technical barriers, and ethical considerations in domain-specific tasks.

  • LLMs offer new methods across hypothesis generation, experimentation, scientific writing, and peer reviewing.
  • The survey identifies opportunities and challenges that shape LLMs’ current capabilities and limitations in scientific research.
  • Continued advances in LLM capabilities may accelerate discoveries and foster innovation and collaboration in scientific workflows.

Limitations

The survey narrows “AI for Science” to LLMs for scientific research. It may omit relevant work by researchers outside computer science and studies not published in computer science venues.

  • Limitations: The survey focuses on LLMs for scientific research rather than the broader “AI for Science” topic.
  • Limitations: Relevant studies by researchers without a computer science background may be absent.
  • Limitations: Work not published in computer science venues may also have been missed.
Loading 2501.04306v1…