PaperlayerView exact revision on arXiv

Source-linked AI summary

MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses

Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, Dongzhan Zhou

arXiv:2410.07076v6cs.CLcs.AIcs.LG

TL;DR

Whether LLMs can discover novel, valid chemistry hypotheses from a research background remains unclear. The paper decomposes discovery into inspiration retrieval, hypothesis composition, and ranking, and implements these steps in MOOSE-Chem. On a challenging benchmark, the framework rediscovered many hypotheses with high similarity while capturing their main innovations.

  • Problem

    It remains unclear whether LLMs can discover novel and valid hypotheses in natural-science domains such as chemistry.

    Source: §1 · p. 2

  • Method

    The paper decomposes chemistry hypothesis discovery into retrieving inspirations, composing hypotheses from background and inspirations, and ranking them in a multi-agent LLM framework.

    Source: §1 · pp. 2–3, §4.2.1 · p. 5

  • Results

    MOOSE-Chem rediscovered many hypotheses with high similarity to the ground truth, covering their main innovations in a setting with up to 3,000 chemistry papers for inspiration selection.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors propose that LLMs may encode knowledge associations that scientists have not yet recognized.

    Source: §1 · p. 3

    The authors caution that LLMs may not be reliable enough to directly evaluate chemistry hypotheses, so their scores serve as preliminary quality indicators for ranking.

    Source: §4.2.4 · p. 7

Abstract

from arXiv · show

Scientific discovery plays a pivotal role in advancing human society, and recent progress in large language models (LLMs) suggests their potential to accelerate this process. However, it remains unclear whether LLMs can autonomously generate novel and valid hypotheses in chemistry. In this work, we investigate whether LLMs can discover high-quality chemistry hypotheses given only a research background-comprising a question and/or a survey-without restriction on the domain of the question. We begin with the observation that hypothesis discovery is a seemingly intractable task. To address this, we propose a formal mathematical decomposition grounded in a fundamental assumption: that most chemistry hypotheses can be composed from a research background and a set of inspirations. This decomposition leads to three practical subtasks-retrieving inspirations, composing hypotheses with inspirations, and ranking hypotheses - which together constitute a sufficient set of subtasks for the overall scientific discovery task. We further develop an agentic LLM framework, MOOSE-Chem, that is a direct implementation of this mathematical decomposition. To evaluate this framework, we construct a benchmark of 51 high-impact chemistry papers published and online after January 2024, each manually annotated by PhD chemists with background, inspirations, and hypothesis. The framework is able to rediscover many hypotheses with high similarity to the groundtruth, successfully capturing the core innovations-while ensuring no data contamination since it uses an LLM with knowledge cutoff date prior to 2024. Finally, based on LLM's surprisingly high accuracy on inspiration retrieval, a task with inherently out-of-distribution nature, we propose a bold assumption: that LLMs may already encode latent scientific knowledge associations not yet recognized by humans.

1 INTRODUCTION

The paper asks whether LLMs can discover novel and valid chemistry hypotheses from a research background, addressing uncertainty about their ability in natural science. It decomposes discovery into inspiration retrieval, hypothesis composition, and ranking, then evaluates a multi-agent framework on a benchmark of recent chemistry papers.

1 INTRODUCTION

  • Prior work evaluated hypothesis generation in social science and NLP, but the authors say LLMs’ chemistry discovery ability remained unclear.Earlier catalyst-discovery evaluations focused on rediscovering commercially used catalysts and could be affected by data contamination.Source: §1 · p. 2
  • The paper decomposes hypothesis discovery into retrieving inspirations, composing hypotheses from background and inspirations, and ranking candidate hypotheses.The authors argue these executable subtasks approximate sufficient conditions for the broader discovery question.Source: §1 · p. 2
  • MOOSE-Chem implements the decomposition through inspiration search, hypothesis generation, and ranking, using evolutionary mutations and multi-step inspiration retrieval.Its framework is intended for chemistry and material science, domains the authors say prior methods did not cover.Source: §1 · pp. 2–3
  • MOOSE-Chem rediscovered many hypotheses with high similarity to the ground truth, capturing their main innovations in a challenging setting.The evaluation used only a background and a corpus of up to 3,000 chemistry papers for inspiration selection.Source: §1 · p. 2
  • The authors construct a benchmark of 51 high-impact chemistry papers, annotated by chemistry PhD students with backgrounds, inspirations, and hypotheses.The benchmark also motivates a ranking task, since checking many generated hypotheses individually requires substantial experimental costs.Source: §1 · pp. 2–3
  • The authors report that their LLM-based framework largely rediscovered the main innovations of many published Nature and Science chemistry hypotheses without data contamination.They controlled the LLM training-corpus date and the papers’ online publication dates.Source: §1 · p. 3

2 RELATED WORK

Prior work explores hypothesis generation across social science, NLP, mathematics, biomedical research, and chemistry, alongside systems for scientific experimentation and discovery. The paper distinguishes its focus on broad chemistry hypothesis discovery from narrower domains and evaluations.

2 RELATED WORK

  • Prior scientific-discovery work spans corpus comparison, LLM-generated hypotheses, mathematical conjectures, biomedical discovery, experimental systems, and material prediction.The passage notes that Wang et al. (2024b) found generated NLP and biochemical hypotheses lagging scientific papers in novelty, depth, and utility.Source: §2 · p. 3
  • Yang et al. (2024b) reported expert-evaluated novel and valid hypotheses, but focused on social science rather than chemistry.FunSearch finds specific mathematical-conjecture solutions but does not discover new theorems.Source: §2 · p. 3
  • Sprueill et al. (2023; 2024) focus on catalyst discovery, evaluating whether they can rediscover existing commercially used catalysts; the authors say this evaluation might cause data contamination.Source: §2 · p. 3

3 BENCHMARK CONSTRUCTION

TOMATO-Chem is designed both to assess LLMs on discovery subtasks and to challenge them to rediscover high-impact chemistry hypotheses from research backgrounds. Its 51-paper collection is annotated and checked by chemistry PhD students.

3 BENCHMARK CONSTRUCTION

  • TOMATO-Chem tests LLM abilities on three smaller discovery questions and challenges them to rediscover high-impact hypotheses from only a research background.The setting resembles a research copilot receiving a scientist’s question and optionally a short survey of existing methods.Source: §3 · p. 3
  • Each paper is split into background questions and surveys, one to three inspiration papers with rationales, a hypothesis, experiments, and related reasoning materials.The reasoning records how background and one or more inspirations combine to form the hypothesis.Source: §3 · pp. 3–4
  • Strict background versions make cases harder when a survey or insightful question already hints at the hypothesis’s direction.The authors say this reflects situations where scientists provide comprehensive surveys or deep insights in their questions.Source: §3 · p. 3
  • The benchmark contains 51 top-venue chemistry and material science papers made public online after January 2024, checked by chemistry PhD students.Experts checked inspiration identification, background separation from the hypothesis and inspirations, and whether those components could logically lead to the hypothesis.Source: §3 · p. 4
  • The benchmark includes about 13 papers inside material science, and its tables report category and publication-venue distributions.Material science is treated as a chemistry subcategory, and additional papers intersect with the field.Source: §3 · p. 4, Table 1 · p. 4, Table 2 · p. 4

4 METHODOLOGY

MOOSE-Chem operationalizes the assumption that chemistry hypotheses combine a research background with inspirations. Its LLM framework retrieves inspirations, composes hypotheses, and ranks them.

4.1 FUNDAMENTAL ASSUMPTION AND FOLLOWING DECOMPOSITION

  • Most chemistry hypotheses are assumed to arise from a research background and inspirations, decomposed into iterative inspiration selection and hypothesis composition, then organized as an MDP.The framework also models a ranking function; typically, one to three inspirations are needed for a chemistry hypothesis.Source: §4.1 · pp. 4–5

4.2.1 THE GENERAL PICTURE

  • Given a research question or survey, the framework screens literature for inspirations, composes hypotheses from them, and ranks those hypotheses; later rounds build on earlier results.MOOSE-Chem directly implements the decomposition, with ranked hypotheses as its output.Source: §4.2.1 · p. 5

4.2.2 DESIGN DETAILS OF P(ij | b, hj−1, I) AND ITS MOTIVATION

  • Inspiration retrieval screens literature papers and asks an LLM to select candidates, avoiding citation or semantic neighbors because they may already be related to the background.The authors propose that advanced LLMs may recognize knowledge pairs whose relation is not yet known to scientists.Source: §4.2.2 · p. 6

4.2.3 DESIGN DETAILS OF P(hj | b, hj−1, ij) AND ITS MOTIVATION

  • The evolutionary unit generates distinct ways to associate background and inspiration, refines them using feedback, then recombines the selected variants into a hypothesis.Feedback covers validness, novelty, clarity, and significance.Source: §4.2.3 · p. 6

4.2.4 DESIGN DETAILS OF R(h) AND ITS MOTIVATION

  • LLMs rank hypotheses using validness, novelty, significance, and potential scores; in chemistry, these scores serve as preliminary filters because direct evaluation may be unreliable.The average score determines which hypotheses advance by beam search; the authors keep the rating function simple to save inference time.Source: §4.2.4 · pp. 6–7

5 INVESTIGATION ON FUNDAMENTAL QUESTIONS

The section tests inspiration retrieval, hypothesis composition, and ranking as component tasks in scientific discovery. Results indicate strong inspiration retrieval, promising hypothesis generation, and a qualified ability to rank hypotheses.

5.1 HOW WELL CAN LLMS PERFORM P(ij | b, hj−1, I)?

  • More than 75% of ground-truth inspirations were covered by the 4% of the chemistry literature corpus selected.The reported result supports the possibility that LLMs associate knowledge pairs not yet recognized by scientists as related.Source: §5.1 · p. 7
  • 83.7% Hit Ratio came from selecting 4% of the corpus with 15-paper windows over two rounds, versus 71.6% at 5% with 60-paper windows over one round.The comparison is reported as evidence that a smaller screening window can perform better.Source: §5.1 · p. 8
  • Inspiration retrieval appears at small model scales and then quickly plateaus.The comparison evaluates Llama models and GPT-4o; the supplied table caption specifies a corpus of 300 and selection of 3 papers per 15-paper window.Source: §5.1 · p. 8, Table 5 · p. 8

5.2 HOW WELL CAN LLMS PERFORM P(hj | b, hj−1, ij)?

  • Because cross-topic chemistry hypotheses can be difficult for researchers to judge reliably, the study uses a six-point Matched Score comparing generated and original hypotheses.For the evaluation, MOOSE-Chem receives only the ground-truth inspiration papers and searches for k rounds before generating hypotheses; experts assess the top two GPT-4o-scored hypotheses per background, and surveys can assist discovery.Source: §5.2 · p. 8

5.3 HOW WELL CAN LLMS PERFORM R(h)?

  • More matched inspirations were associated with better average ranking ratios, and higher Matched Scores generally aligned with better ranks for scores from 2 to 4.The ranking evidence is qualified: the disadvantage of hypotheses without a positive Matched Score was not large, and some may still have been high quality.Source: §5.3 · p. 9

6 EXPERIMENT AND ABLATION STUDY

The experiments evaluate MOOSE-Chem in a copilot-like setting and compare it with baselines and ablations. Ablations show gains in the best-performing hypothesis from mutation and recombination and multi-step design, while expert evaluation assesses hypotheses produced in the real-copilot setting.

6 EXPERIMENT AND ABLATION STUDY

  • The copilot-style experiment supplied a strict background question or survey and a 300-paper corpus, selecting only the top 4% for hypothesis development.GPT-4o had training data through October 2023; results were evaluated using Top MS and Average MS averaged across the benchmark.Source: §6 · p. 9

6.1 BASELINES

  • MOOSE is a social-science baseline using self-refinement but no mutation/recombination or multi-step inspiration; SciMON and Qi et al. retrieve information by semantic/citation neighbors or background keywords.The paper notes these retrieval approaches may return material related to the background rather than information serving as an inspiration.Source: §6.1 · pp. 9–10

6.2 RESULTS

  • Mutation and recombination and multi-step design each improved the best-performing hypothesis, while mutation and recombination reduced Average MS versus the MOOSE baseline.The authors attribute the Average MS decrease to mutation sometimes producing hypotheses that make little sense and receive low Matched Scores.Source: §6.2 · p. 10, Table 10 · p. 10
  • In the real copilot setting, two experts evaluated each background’s four generated hypotheses with the highest GPT-4o MS; an expert MS of 3 is already very high and indicates coverage of two main innovations.Source: §6.2 · p. 10

7 CONCLUSION

The conclusion presents a decomposition of chemistry hypothesis discovery into three practical questions and a multi-agent framework implementing them. In a copilot-in-the-wild setting, the framework rediscovered many hypotheses with high similarity to ground truth and captured their main innovations.

7 CONCLUSION

  • The proposed multi-agent framework implements three smaller questions derived from a fundamental assumption and is evaluated on chemistry and material-science papers published and first made public in 2024.The experiments use LLMs trained through October 2023.Source: §7 · p. 10
  • In the copilot-in-the-wild setting, MOOSE-Chem rediscovered many ground-truth hypotheses with very high similarity, covering their main innovations.The framework was given only a research background and used LLMs trained through October 2023.Source: §7 · p. 10

A.1 MOOSE-CHEM OVERVIEW I/O FIGURE

Figure 2 is identified as an overview of MOOSE-Chem’s input and output. The accompanying text says that titles and abstracts of numerous papers form the inspiration search space.

A.1 MOOSE-CHEM OVERVIEW I/O FIGURE

  • Figure 2 presents an overview of the framework’s input and output.The passage does not specify the figure’s individual visual encodings.Source: Figure 2 · p. 15
  • The inspiration corpus is initialized from paper titles and abstracts, which together form the search space.Source: §A.1 · p. 15

A.2 PROOF OF THE RIGOROUS DECOMPOSITION OF P(h | b) BASED ON THE FUNDAMENTAL ASSUMPTION

The decomposition formalizes the assumption that chemistry hypotheses can be built from research backgrounds and inspirations, then treats discovery as sequential inspiration selection and hypothesis construction. It makes this process tractable through simplifying assumptions and a Markov decision-process formulation.

A.2 PROOF OF THE RIGOROUS DECOMPOSITION OF P(h | b) BASED ON THE FUNDAMENTAL ASSUMPTION

  • Unlike Swanson Linking, the assumption generally allows multiple inspirations, does not require intermediate-paper paths, and applies to a majority of published chemistry hypotheses.The authors suggest that future chemistry hypotheses may similarly arise from linkages among existing literature.Source: §A.2 · p. 15
  • Most chemistry hypotheses can be composed from a research background and additional knowledge elements called inspirations, an assumption supported by researcher discussions and cognitive-science findings.The paper presents identifying and integrating suitable inspirations with background knowledge as necessary and sufficient for generating a valid hypothesis.Source: §A.2 · p. 15
  • Inspiration retrieval is inherently out-of-distribution because useful connections must not already be established or used to formulate hypotheses.Retrieved papers may contain noisy, extraneous information, while one inspiration may appear across multiple papers; that redundancy can help recover it despite imperfect searches.Source: §A.2 · p. 16
  • The derivation assumes each hypothesis has a unique minimal inspiration set and adopts a fixed canonical order, reducing the probability over inspiration sequences to a chain-rule factorization.Chemistry inspirations may be largely interchangeable, whereas mathematics may require more constrained ordering.Source: §A.2 · pp. 15, 17, 18
  • The full inspiration space includes potentially relevant knowledge from other disciplines, but the implementation approximates it with about 3,000 highly cited chemistry papers.Computing over the full space is described as computationally infeasible.Source: §A.2 · p. 17
  • Because researchers integrate components incrementally, the framework recursively builds intermediate hypotheses, with each update depending on the background, prior hypothesis state, and current inspiration.This Markov structure represents hypothesis states as states and inspirations as actions in a sequential decision process.Source: §A.2 · p. 17
  • The resulting decomposition models each step as selecting an inspiration and updating the intermediate hypothesis, with hypothesis quality or plausibility serving as reward.The formulation breaks the overall discovery probability into practical, executable questions.Source: §A.2 · pp. 18–19

A.3 THE FULL INSTRUCTION FOR BENCHMARK CHECKING

The benchmark-checking instructions ask reviewers to verify that each paper’s background, inspirations, and hypothesis are correctly identified and logically connected.

A.3 THE FULL INSTRUCTION FOR BENCHMARK CHECKING

  • Check that the background question is correct, appropriately general, and specific enough to identify its research domain without embedding the inspiration or hypothesis.The survey should not contain inspiration or hypothesis information unless that information was previously used for the particular background question.Source: §A.3 · p. 19
  • Include the most similar existing method in the background survey, but exclude inspiration elements that have not already been used for the question.The instructions illustrate this boundary with a proposed BaCl2-to-BaSO4 change: BaCl2 may appear in the survey, but SO4 belongs to the inspiration.Source: §A.3 · p. 19
  • Verify that the identified inspirations are the paper’s main inspirations and add any missing main inspiration.Source: §A.3 · p. 19
  • Check that the main hypothesis is correct and covers its key points.Source: §A.3 · p. 19
  • Ensure the background survey, background question, and identified inspirations can logically lead to the hypothesis.If they cannot, the instructions suggest identifying additional inspirations.Source: §A.3 · p. 19

A.4 PROMPT TO OBTAIN R(h)

The prompt scores research hypotheses on validity, novelty, significance, and potential, each using a five-point scale. It also describes matched-score summaries and an ablation-study evaluation.

A.4 PROMPT TO OBTAIN R(h)

  • Validity scores whether a hypothesis follows current research and theory, has feasible experimental verification, and is supported by evidence, with lower scores for implausible conditions or contradictions.The five-point rubric ranges from strongly supported and straightforward to test to unsupported by theory with no clear experimental path.Source: §A.4 · p. 20
  • Novelty scores how far a hypothesis departs from established norms, ranging from minor applications of known principles to potentially field-altering ideas.Source: §A.4 · p. 20
  • Significance scores the potential scale of contribution, from narrow confirmation to changes in a branch of chemistry.Source: §A.4 · p. 20
  • Potential scores how much a hypothesis could contribute if enhanced with an additional methodological component, from limited added value to possible transformative advances.Source: §A.4 · pp. 20–21
  • The results tables report benchmark statistics for average and highest matched scores, while the experiments and ablation table uses Claude-3.5-Sonnet to evaluate matched score.The table caption defines Average/Top MS as the average/highest Matched Score among hypotheses generated from one background.Source: Table 12 · p. 21, Table 13 · p. 21

A.5 AUTOMATIC EVALUATION BY CLAUDE AND GEMINI

Alternative evaluators support the reported robustness of MOOSE-Chem and its components, while appendix analyses identify limits in ranking and show that matched-score agreement is not a complete measure of hypothesis quality. The discussion also frames discovery as proposing unseen but potentially valid hypotheses through associations among knowledge.

A.5 AUTOMATIC EVALUATION BY CLAUDE AND GEMINI

  • Evaluations with Claude-3.5-Sonnet and Gemini-1.5-Pro report robust results for MOOSE-Chem and its components, consistent with LLMs associating known knowledge into potentially valid unknown knowledge.Table 13 and Table 14 re-evaluate the hypotheses from Table 10; Table 14 uses Gemini-1.5-Pro for Matched Score.Source: §A.5 · pp. 21–22, Table 14 · p. 22

A.6 MORE ANALYSIS ON EU

  • About one-third of high-quality hypotheses came directly from the non-EU branch, while the recombination branch contained more; omitting significance feedback could improve Matched Score.The authors suggest that prompting for significance may lead LLMs to generate more divergent hypotheses, which can lower similarity to a given reference.Source: §A.6 · p. 22, §A.7 · p. 22, Table 16 · p. 22

A.8 RANKING OF GROUND TRUTH HYPOTHESES

  • Ground-truth hypotheses were not ranked at the top; proposed explanations include weak LLM ranking, self-descriptions affecting judgments, or generated hypotheses surpassing or elaborating on the originals.The authors also note that more detail may result from iterative clarity feedback and refinement.Source: §A.8 · p. 23, Table 17 · p. 23

A.9 INFLUENCE OF RESEARCH BACKGROUND OPTIONS TO INSPIRATION RETRIEVAL

  • A survey can substantially help inspiration retrieval, while removing the strict-background condition slightly lowers Hit Ratio, which the authors attribute to background information discouraging retrieval of related inspirations.Table 18 compares background options using a corpus of 300 papers and screening windows of 15 papers, with 3 selected per window.Source: §A.9 · p. 23, Table 18 · p. 23

A.10 DISCUSSION ON HALLUCINATION AND SCIENTIFIC DISCOVERY

  • The discussion treats hallucination as a source of novel hypotheses, situates discovery in inductive reasoning, and distinguishes retrieving previously unrecognized relations from retrieving semantically similar material.The authors contrast this discovery-oriented retrieval with conventional RAG and assume LLMs may identify such relations.Source: §A.10 · p. 23, §A.11.1 · pp. 23–24, §A.11.2 · p. 24

A.11.3 SELF CONSISTENCY

  • Evolutionary units explore diverse options for selection rather than voting for consistency; Matched Score compares methodological key points, and a low score does not establish poor hypothesis quality.The authors say only laboratory experiments can check each hypothesis.Source: §A.11.3 · p. 24, §A.12 · p. 24, §A.13 · p. 25

A.14 EVALUATION AGREEMENT BETWEEN EXPERT EVALUATION AND GPT-4O EVALUATION

  • Automatic and expert Matched Score evaluations show medium-to-high consistency, with GPT-4o scores usually 1–2 points higher; expert evaluations show high consistency with one another.The authors suggest experts may judge some generated hypotheses insufficiently clear for communicating innovations to scientists.Source: §A.14 · p. 25, §A.15 · p. 25, Table 19 · p. 25

A.16.1 CASE 1 (MS BY EXPERT: 3)

  • Expert case reviews found useful matches alongside unsupported additions: nitrogen doping was suggestive, hydrogel freeze-casting aligned with the reference, and some ligand or solvent proposals differed from the studies.The case analyses also identify proposed additions as possible avenues for further investigation.Source: §A.16.1 · p. 26, §A.16.2 · p. 26, §A.16.3 · p. 27
Loading 2410.07076v6…