Large Language Models are Zero Shot Hypothesis Proposers

1 Introduction

• Expanding literature and disciplinary silos impede knowledge discovery, while formal tools for generating novel hypotheses remain underexplored.The introduction also identifies source traceability as a challenge for exploring zero-shot hypothesis generation.Source: §1 · p. 2
• LLMs generated hypotheses absent from training data that were validated against test literature, showing rudimentary higher-order reasoning.The authors report this across model and scenario evaluations.Source: §1 · pp. 2–3
• Increasing uncertainty can diversify candidates and potentially improve zero-shot generation, while few-shot enhancements and added tools may reduce proficiency.The authors note that heightened uncertainty in collaborative multi-agent approaches significantly improves zero-shot generalization.Source: §1 · p. 2
• The authors build temporal biomedical instruction data and design experiments to test zero-shot and few-shot hypothesis generation, claiming this is the first formal investigation of the capacity.The training literature predates January 2023, and the unseen test set uses August 2023 literature while evaluated models are restricted to earlier training corpora.Source: §1 · pp. 2–3
• Four evaluation dimensions support ChatGPT-based and human reviews, and correlations with manual results suggest LLMs can also contribute to hypothesis evaluation.Source: §1 · p. 3
• An LLM-based multi-agent system uses collaboration and tools to explore factors affecting hypothesis generation.Source: §1 · p. 3

2 Process of Scientific Discovery

• The discovery loop analyzes past observations and data to generate hypotheses, which guide experiments that produce data for the next cycle.Source: Figure 2 · p. 3, §2 · p. 3
• The process comprises data analysis, hypothesis generation, experiment design and execution, then observation and reflection that can support or prompt hypothesis revision.Source: §2 · p. 3
• Hypothesis formulation has a distinctive role in steering research-question selection, experiment design, and data analysis.Source: §2 · p. 3

3.1 Problem Definition

• The task treats hypothesis generation as completing a hypothesis from an instruction and biomedical background knowledge.The objective models each hypothesis token conditioned on previous tokens, the instruction, and background.Source: §3.1 · p. 4

3.2 Dataset Construction

• The dataset separates training literature published before January 2023 from post-January test literature, with August 2023 papers used for the unseen test set.The authors state that models were selected to ensure no prior exposure to the test data.Source: §3.2 · p. 4

3.3 Datast Analysis

• The unseen dataset contains 200 background–hypothesis pairs from August 2023; the seen dataset contains 2,700 pairs from before January 2023.The seen data are partitioned into 2,500 training and 200 validation examples, as well as a seen test set.Source: §3.3 · p. 4

3.5.1 Impact of zero-shot settings

• Zero-shot prompts improve hypothesis generation, particularly novelty; for some large models, randomly sampled few-shot examples perform worse.The reported examples include Llama-2-70b-chat and WizardLM-70B-V1.0.Source: §3.5.1 · pp. 5–6
• LLMs tend to perform better on the unseen than the seen test set, despite possible memorization of hypotheses in pretraining.The authors speculate that hypothesis complexity may hinder models from using stored knowledge effectively.Source: §3.5.1 · p. 6

3.5.2 Influence of external knowledge

• Few-shot examples increase verifiability but reduce novelty, while similarity-retrieved examples further improve performance over randomly sampled examples.The paper also reports that external knowledge does not consistently improve hypothesis proposing.Source: §3.5.2 · p. 6
• Fine-tuning WizardLM-13B-V1.2 on background–hypothesis pairs yields superior performance to GPT-3.5-turbo and WizardLM-70B-V1.0.Domain-adapted MedAlpaca and PMC-LLaMA improve word overlap, but ChatGPT evaluations indicate only minimal effects, motivating more unified metrics.Source: §3.5.2 · p. 7

3.6.2 Evaluation Results

• Human and ChatGPT evaluations have Pearson and Spearman correlations above 0.7 for all evaluated models.Human evaluation covered randomly selected unseen-test examples scored by three evaluators with biomedical backgrounds.Source: §3.6.2 · p. 7, §3.6.1 · p. 7

4.1 Multi-agent Framework

• The framework uses four automated agents with optional human involvement in an iterative hypothesis-formulation process.The Analyst extracts core elements, the Engineer retrieves information, the Scientist crafts a hypothesis, and the Critic evaluates it.Source: §4.1 · pp. 8–9, Figure 4 · p. 8
• The Engineer can use PubMed and other search engines, with post-January 2023 literature filtered from results to protect test-set visibility.The experiments use ReAct or OpenAI function calling for tool use.Source: §4.1 · p. 9

4.2 Experiment Results

• Tool use has minimal impact on hypothesis generation, and ReAct performs worse than OpenAI function calling.The authors report that models struggle to identify useful information and to use the thought–action–observation process effectively.Source: §4.2 · p. 9
• Multi-agent division of labor and interaction enhance hypothesis proposing by introducing uncertainty; tool use can further improve the multi-agent framework.The authors relate the interaction to iterative discussions and refutations in scientific research.Source: §4.2 · p. 9

5 Conclusion

• The corpus pairs biomedical background knowledge with hypotheses and is split by publication date.The paper identifies effective uncertainty assessment metrics as a focus for future work.Source: §5 · pp. 9–10
• Fine-tuned models of different sizes generate hypotheses absent from training data but confirmed by test literature, with performance comparable to or sometimes better than ChatGPT.The conclusion also reports that introducing uncertainty enhances zero-shot generalization capabilities.Source: §5 · p. 10

A.1 Data-Driven Scientific Discovery

• Prior scientific-discovery datasets and methods have limitations including non-recent discoveries, reliance on human knowledge, or lack of strict control over test-data visibility.The paper contrasts these works with datasets partitioned by publication date to ensure test-data independence from models.Source: §A.1 · p. 13

A.2 LLM-driven Autonomous Agents

• The paper characterizes multi-agent use for scientific discovery as underexplored and presents its work as a preliminary effort to enhance hypothesis proposing with tools and agents.The discussion describes autonomous agents as planning sub-goals, executing actions, receiving environmental feedback, and adjusting behavior.Source: §A.2 · p. 13

• The implementation details cover the constructed dataset and the open-source models used in the experiments.The passage does not specify further details here.Source: B Implementation Details · p. 13

B.1 Details of Dataset

• Figure 5 compares publication-date and topic distributions across the training and test datasets.Training and seen-test publications increase year by year through January 2023; the unseen set samples August 2023 and emphasizes its latter part.Source: §B.1 · p. 13, Figure 5 · p. 14

B.2 Details of Models

• Table 4 summarizes model training corpora and publication dates to assess whether unseen-test data was visible during training.The table also covers pre-training and supervised-learning corpora.Source: Table 4 · p. 14, §B.2 · p. 14

C Additional Results

• ChatGPT evaluations correlate more highly with human evaluations than BLEU and ROUGE-L, while word-overlap metrics are weak measures of generation quality.BLEU and ROUGE-L can correlate highly with relevance, but do not comprehensively evaluate generated hypotheses; the section points to Table 5 for correlation scores.Source: C Additional Results · p. 14, Table 5 · p. 15

D Case Study

• The case study presents generated hypotheses from different models and walks through the ChatGPT evaluation process.Source: D Case Study · p. 15

MULTI-AGENT CASE

• The multi-agent case uses Analyst, Scientist, and Critic roles across six rounds to develop a hypothesis about telemonitoring and wearable cardiac monitoring.The table presents this collaboration case.Source: §D.1 · p. 15, Table 7 · p. 17

D.1 Generated Hypothesis

• gpt-3.5-turbo (0-shot) proposes the power-law-index hypothesis from background alone for an August 2023 myocardial-infarction paper.The paper describes this as an example of a highly novel and verifiable hypothesis proposal.Source: Table 6 · p. 15, §D.1 · p. 15
• The gpt-3.5-turbo zero-shot proposals include αshort and αlong as potential biomarkers for early myocardial-infarction detection.The listed proposals also discuss E2, E3, Etotal, and two-stage power-law rheology in cardiac tissue.Source: §D.1 · p. 15

MULTI-AGENT CASE

• The Critic asks for more specific patient outcomes, mechanisms linking monitoring to outcomes, and references supporting the hypothesis.Source: MULTI-AGENT CASE · p. 19
• The final hypothesis specifies possible outcomes and mechanisms, including reduced hospitalizations, early abnormal-rhythm detection, timely interventions, and personalized treatment plans.It also says the hypothesis is supported by studies on telemonitoring and detection of actionable events.Source: MULTI-AGENT CASE · p. 20

Evaluation Examples

• Novelty received a score of 2 because the hypothesis proposes mechanisms and insights beyond the background on telemonitoring and wearable monitoring.The explanation points to earlier detection, timely intervention, and personalized treatment as proposed mechanisms.Source: §D.2 · p. 21
• Relevance received a score of 3 because the hypothesis builds on the supplied background about cardiac-device telemonitoring and wearable monitoring.The explanation characterizes the proposed patient outcomes as logically related to that background.Source: §D.2 · p. 22
• Significance received a score of 3, with the explanation describing the hypothesis as testable and as extending beyond the background.The cited background includes evidence that telemonitoring can reduce time to detection of clinically actionable events and improve patient outcomes.Source: §D.2 · p. 22
• The hypothesis links telemonitoring and wearable cardiac monitoring to improved patient outcomes through earlier detection, timely intervention, and personalized treatment.The proposed outcomes include reduced hospitalizations, improved quality of life, and decreased mortality.Source: §D.2 · p. 22
• Verifiability received a score of 3 because the explanation cites studies on telemonitoring outcomes and improved detection of actionable events.ChatGPT provides a step-by-step explanation and integer scores for each metric.Source: §D.2 · pp. 21–22

E Prompts Design

• The prompt appendix includes materials for hypothesis generation and for ChatGPT’s evaluation process.The accompanying Table 8 is described as a case illustrating ChatGPT-based hypothesis evaluation.Source: E Prompts Design · p. 22

E.1 Zero- and Few-shot Prompts

• Zero-shot and few-shot generation prompts are provided, with few-shot examples obtained through random sampling or similarity retrieval.Table 10 contains manually constructed background-hypothesis examples sampled from literature published before January 2023.Source: §E.1 · p. 23, Table 9 · p. 23, Table 10 · p. 24

E.2 Prompts for Multi-agent Collaboration

• Multi-agent collaboration prompts specify role designs and environment settings.The appendix identifies separate tables for role prompts and environment settings.Source: §E.2 · p. 23, Table 12 · p. 26, Table 11

E.3 Prompts for ChatGPT Evaluation

• ChatGPT evaluation prompts are separately provided for novelty, relevance, significance, and verifiability.The appendix assigns one prompt table to each metric.Source: §E.3 · p. 23, Table 13 · p. 26, Table 14 · p. 26

E.2 Prompts for Multi-agent Collaboration

• The Analyst distills or revises background keywords, while the Engineer searches iteratively and organizes relevant sources into structured clues.Engineer outputs can include source titles, summaries, key findings, implications, and relevant excerpts.Source: §E.3 · p. 25
• The Scientist drafts a scientifically grounded hypothesis from the Engineer’s findings, and the Critic assesses it and can request refinement.When the hypothesis meets the desired standards, the prompt calls for an approved structured answer.Source: §E.3 · p. 25