PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models for Automated Open-domain Scientific Hypotheses Discovery

Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, Erik Cambria

arXiv:2309.02726v3cs.CLcs.AI

TL;DR

Prior hypothetical-induction research used manually selected observations and mostly commonsense hypotheses, rather than open-domain raw web input and hypotheses potentially new to humanity. This paper introduces TOMATO and the MOOSE multi-module framework with three feedback mechanisms. Evaluations report improvement over an LLM-based baseline, and experts judged many outputs both novel and valid.

  • Problem

    Prior hypothetical-induction datasets use manually selected observations and mostly commonsense hypotheses, rather than raw-web observations and hypotheses new to humanity.

    Source: §1 · p. 1

  • Method

    The paper introduces TOMATO and constructs a social-science dataset alongside MOOSE, an LLM-prompting framework with past-, present-, and future-feedback.

    Source: §1 · pp. 1–2, §4 · p. 3

  • Results

    GPT-4 and expert evaluations find MOOSE outperforms an LLM-based baseline, and experts recognize many generated hypotheses as novel and valid.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The authors conclude that LLMs can potentially serve as scientist copilots during hypothesis formation, the first step of scientific discovery.

    Source: §7 · p. 9, Limitations · p. 10

    The dataset comprises 50 recent social-science papers, each manually collected and annotated in detail by experts.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Hypothetical induction is recognized as the main reasoning type when scientists make observations about the world and try to propose hypotheses to explain those observations. Past research on hypothetical induction is under a constrained setting: (1) the observation annotations in the dataset are carefully manually handpicked sentences (resulting in a close-domain setting); and (2) the ground truth hypotheses are mostly commonsense knowledge, making the task less challenging. In this work, we tackle these problems by proposing the first dataset for social science academic hypotheses discovery, with the final goal to create systems that automatically generate valid, novel, and helpful scientific hypotheses, given only a pile of raw web corpus. Unlike previous settings, the new dataset requires (1) using open-domain data (raw web corpus) as observations; and (2) proposing hypotheses even new to humanity. A multi-module framework is developed for the task, including three different feedback mechanisms to boost performance, which exhibits superior performance in terms of both GPT-4 based and expert-based evaluation. To the best of our knowledge, this is the first work showing that LLMs are able to generate novel (''not existing in literature'') and valid (''reflecting reality'') scientific hypotheses.

1 Introduction

Hypothetical induction moves from observations to explanatory hypotheses, but prior datasets used manually selected observations and mostly commonsense hypotheses. The paper introduces an open-domain social-science task and MOOSE, a prompting framework with three feedback mechanisms, reporting stronger evaluations and hypotheses judged novel and valid.

1 Introduction

  • Prior hypothetical-induction datasets rely on manually selected observations and mostly commonsense hypotheses, limiting their fit to raw-web input and novel discovery.The proposed TOMATO setting instead asks systems to generate novel, valid, and helpful research hypotheses from raw web corpus.Source: §1 · p. 1
  • TOMATO requires automatic hypothesis proposals from open-domain raw web data, with few human efforts and no manually collected input data.The dataset contains 50 recent social-science papers; experts assemble related web passages as benchmarking material, while systems must use only the raw corpus.Source: §1 · p. 2
  • MOOSE uses LLM prompting in a multi-module framework with present-, past-, and future-feedback to refine generated hypotheses.Present-feedback refines a generation using evaluation feedback; past-feedback applies later feedback to an earlier module, while future-feedback provides justifications or initial suggestions for later modules.Source: §1 · p. 2
  • GPT-4 and expert evaluations find MOOSE outperforms an LLM-based baseline, while each feedback mechanism progressively improves the base framework.Experts also recognize many MOOSE hypotheses as both novel and valid.Source: §1 · p. 2

2 Related Work

Prior scientific-discovery systems differ from TOMATO in their input requirements, target discoveries, or domain. Existing self-feedback work focuses on present-feedback, whereas this paper also develops past- and future-feedback for inductive reasoning.

2.1 NLP Methods for Scientific Discovery

2.2 LLM-based Self Feedback

  • Self-refine (Madaan et al., 2023) and related approaches focus on present-feedback, unlike this framework's past- and future-feedback mechanisms.The passage describes those approaches as not specially designed for inductive reasoning tasks.Source: §2.2 · p. 2
  • Yang et al. (2024) use LLMs to filter candidate generations, while this paper's present-feedback also provides feedback for refinement.The passage says the earlier approach filters generations but does not provide feedback.Source: §2.2 · p. 3

3 Dataset Collection

The dataset pairs recent social-science hypotheses with expert-identified backgrounds, inspirations, and semantically similar web passages. Its construction preserves a raw-web input setting while recording the reasoning and complexity of each example.

3 Dataset Collection

  • The dataset covers 50 papers published after January 2023, and Table 1 reports their subject distribution.Each paper is from a top social-science journal.Source: §3 · p. 3, Table 1 · p. 3
  • Experts identify each paper's main hypothesis, research background, and theory-based inspirations, then abstract the reasoning from those inputs to the hypothesis.The example's background concerns facial-recognition-payment usage, while a herding-effect theory subsection supplies an inspiration.Source: §3 · p. 3
  • Semantically similar web texts substitute for paper text as the ground-truth background and inspiration, helping avoid contamination and preserving the raw-web task setting.The example uses news sentences about facial-recognition payment and a Wikipedia description of the herding effect.Source: §3 · p. 3
  • The dataset records paper links and dates, includes 14 related-field survey papers for novelty checks, and is constructed by a social-science PhD student.The authors state that they explain why automatic methods should not collect the dataset in §A.4.Source: §3 · p. 3

4 Methodology

MOOSE is a serial, multi-module framework built around the stages of social-science hypothesis formation. Three feedback mechanisms let modules revise outputs using immediate evaluations, later evaluations, or information prepared for future modules.

4 Methodology

  • MOOSE combines a serial multi-module base framework with past-, present-, and future-feedback; the version without feedback is MOOSE-base.The framework is described in Figure 4 and Algorithm 1.Source: §4 · p. 3

4.1 Base Framework

  • MOOSE-base finds research backgrounds, searches corpus titles for inspirations, retrieves matching passages, and uses them with the background to propose hypotheses.The inspiration-title search narrows the search before finding inspirational sentences, avoiding a full-corpus search for every background.Source: §4.1 · p. 4

4.2 Present-Feedback

  • Present-feedback evaluates an output and feeds the output plus feedback back to its module for regeneration, with checks for reality, novelty, and clarity.The framework omits a separate consistency check to save computing power because reality feedback usually covers consistency; clarity feedback also checks whether enough detail is provided.Source: §4.2 · pp. 4–5

4.3 Past-Feedback

  • Past-feedback waits until a later module's output can be evaluated, then uses that feedback to revise the earlier output.It is applied to the Inspiration Title Finder because inspirations are difficult to assess directly and can lead to low-quality hypotheses.Source: §4.3 · p. 5

4.4 Future-Feedback

  • Future-feedback either supplies a module's justifications to later modules or inserts a preliminary suggestion module before a complex task.MOOSE uses the first design for the Background Finder and Inspiration Title Finder, and the suggestion module before the Hypothesis Proposer.Source: §4.4 · p. 5

5 Experiments

The experiments use GPT-4 and expert evaluations to compare MOOSE with an LLM baseline and test its feedback mechanisms. Across these evaluations, the feedback mechanisms generally improve performance, though GPT-4 and expert ratings differ on past-feedback.

5.1 Evaluation Metrics & Details

  • Evaluations score hypotheses for validness, novelty, and helpfulness; experts rate 400 randomly selected hypotheses without knowing which methods produced them.The metrics use a five-point Likert scale, and the paper prioritizes novelty over validness because hypotheses are intended to assist researchers rather than enter the literature directly.Source: §5.1 · pp. 5–6

5.2 Baselines & Base Model Selection

  • The corpus-chunk baseline directly generates hypotheses, while MOOSE modules and the baseline use GPT-3.5-turbo except in the separate Claude3-Opus model comparison.For GPT-4 evaluation, about 1,000 hypotheses are assessed per MOOSE-related method; expert evaluation generally samples 50 per method.Source: §5.2 · p. 6, §5.3 · p. 6

5.3 Main Results

  • MOOSE-base outperforms the baseline on novelty and helpfulness in GPT-4 and expert evaluations, while future-feedback improves MOOSE-base across all three metrics.With GPT-4, adding both future- and past-feedback further improves novelty but slightly lowers validness and helpfulness relative to future-feedback alone; experts find that combination comprehensively better.Source: §5.3 · pp. 6–7, Table 2 · p. 6, Table 4 · p. 6
  • More present-feedback iterations steadily raise GPT-4-rated validness and novelty, while helpfulness peaks at three iterations.Expert-rated performance generally rises with additional iterations, though the results suggest there may be an optimal number.Source: Table 3 · p. 6, Table 5 · p. 6, §5.3 · p. 7

6 Analysis

The analysis examines background and inspiration selection, feedback ablations, base-model choice, and a generated example. Results show trade-offs among validness, novelty, and helpfulness, while MOOSE’s components remain effective with Claude3-Opus.

6.1 Background and Inspirations

  • ChatGPT-selected backgrounds and inspirations score highest in helpfulness among unannotated methods, while random selection can yield comparable validness and novelty.BM25 selection produces higher novelty than ChatGPT selection; ground-truth hypotheses score highest in novelty and helpfulness.Source: §6.1 · p. 7, Table 6 · p. 7

6.2 More Ablation Studies

  • Without FF2, performance drops across metrics; without FF1, validness and novelty fall while helpfulness remains comparable.The authors find FF2 more significant, while adding background or inspiration prospects directly to the hypothesis proposer yields more valid but much less novel hypotheses.Source: §6.2 · pp. 7–8, Table 7 · p. 7
  • Without access to related surveys, novelty rises by 0.04 while validness falls to around 0.26.The authors attribute this partly to BM25 retrieving too few similar survey chunks, which can lead the novelty detector to judge hypotheses novel because they are absent from retrieved surveys.Source: Table 7 · p. 7, §6.2 · p. 8
  • Using the inspiration corpus for background extraction raises validness by about 0.025 but lowers novelty by about 0.16.The authors suggest that reusing an inspiration passage for both background and inspiration retrieval can reduce novelty.Source: Table 7 · p. 7, §6.2 · p. 8

6.3 Effect of Base Model Selection

  • MOOSE’s components remain effective with Claude3-Opus, and all three evaluation scores improve substantially over GPT-3.5.The authors interpret the stronger scores as evidence of greater framework potential with more powerful LLMs.Source: §6.3 · p. 8, Table 8 · p. 8, Table 9 · p. 8

6.4 Qualitative Analysis

  • One expert-rated hypothesis proposes that collectivist cultures engage in more conspicuous consumption than individualistic cultures, scoring 3.3 for validness and 4.0 for both novelty and helpfulness.The expert describes it as counter-intuitive relative to popular perceptions and connects its validity to cultural theories and empirical evidence.Source: §6.4 · pp. 8–9

7 Conclusion

The paper introduces TOMATO, an open-domain task for social-science hypothesis discovery, and presents MOOSE as a multi-module framework with feedback mechanisms. Its experiments report that MOOSE can generate hypotheses experts recognize as both novel and valid, supporting its potential as a scientist copilot.

7 Conclusion

  • The paper introduces TOMATO, the first NLP task focused on discovering social-science research hypotheses, and constructs a dataset of 50 recent papers from top academic journals.It also develops MOOSE, a multi-module framework with three feedback mechanisms.Source: §7 · p. 9
  • MOOSE-base outperforms an LLM-based baseline, and the three feedback mechanisms further improve its performance.The conclusion reports that PhD-student evaluators recognized many MOOSE-generated hypotheses as both novel and valid.Source: §7 · p. 9
  • The authors present MOOSE as evidence that LLMs have potential to serve as a copilot for scientists generating novel and valid hypotheses.They describe the hypotheses as novel when not existing in the literature and valid when reflecting reality.Source: §7 · p. 9

Limitations

The authors discuss the dataset’s limited size and the framework’s uncertain transfer beyond social science, while noting its expert annotations and disciplinary coverage. They also argue that automated generation can support controllable generation and describe MOOSE’s potential research use.

Limitations

  • The dataset contains only 50 recent papers, but experts annotate them with detailed background, inspiration, and related web-passage information.The authors describe the papers as appearing in top social science journals.Source: Limitations · p. 9
  • The framework’s applicability beyond social science is unclear; the dataset covers seven disciplines, and the authors chose the field because its main data format is language.They describe a general method spanning all disciplines as nearly impossible for the first few works.Source: Limitations · p. 9
  • Although some scientists may prefer controllable generation from their own inputs, the authors argue an automated system can support it by substituting manually crafted inputs.The examples of replaceable inputs are background and inspirations.Source: Limitations · p. 9
  • MOOSE might serve as a social-science research copilot for hypothesis formation, and the authors advocate open-sourcing such systems to broaden access.They describe hypothesis formation as the first step for scientific discovery.Source: Limitations · p. 10

A.1 Hyper-parameters

The appendix specifies which LLMs were used in the experiments and reports the sampling settings for generation and GPT-4 evaluation.

A.1 Hyper-parameters

  • Claude3-Opus is used in §6.3, while all other experiments use gpt-3.5-turbo.Source: §A.1 · p. 12
  • 0.9 temperature and 0.9 top_p are used with both Claude3-Opus and gpt-3.5-turbo.Source: §A.1 · p. 12
  • GPT-4 evaluation uses 0.0 temperature and 0.9 top_p to keep evaluation scores stable.Source: §A.1 · p. 12

A.2 More Related Works on Reasoning and Scientific Discovery

The authors distinguish their focus on novel social-science hypotheses from commonsense and case-based reasoning, and compare their setting with concurrent hypothesis-proposing work.

A.2 More Related Works on Reasoning and Scientific Discovery

A.3 Dataset Complexity Distribution

The appendix examines dataset complexity and the role of expert judgment in open-domain hypothesis construction and evaluation, alongside expert–GPT-4 agreement and qualitative analyses of MOOSE. It also provides a blind-box formulation example, future research directions, and algorithm details.

A.4 Why the Tomato Dataset Shouldn’t Be Collected by Automatic Methods

  • Selecting suitable hypotheses and assembling their dispersed background and inspirations requires expert judgment and deep domain understanding.Table 10’s “Easy” label is relative to other publications in the dataset, not a claim that inducing the hypothesis is actually easy.Source: §A.4 · p. 12, §A.3 · p. 12

A.5 Why Not Using Other Evaluation Metrics

  • The authors reject relevance and significance metrics, and reject BLEU, ROUGE, and METEOR because open-domain hypotheses lack fixed backgrounds and may be novel or differently grounded.They also argue that significance is difficult to compare in social science and varies over time.Source: §A.5 · pp. 12–13

A.6 Hypotheses Selection for Expert Evaluation

  • Experts evaluated 400 hypotheses drawn from 50 backgrounds, with eight hypotheses per background spanning the baseline and MOOSE variants.One expert assessed all 400, while two others each assessed 104; scores were averaged across experts.Source: §A.6 · p. 13

A.7 Expert Qualification and Expert Agreement

  • The evaluators were three Marketing PhD students, and expert ratings showed high agreement, with all soft consistency scores above 0.75 means and average differences below 1 point.The consistency analysis covers Validness, Novelty, and Helpfulness.Source: §A.7 · p. 13, Table 11 · p. 13

A.8 Consistency Between Expert Evaluation and GPT-4 Evaluation

  • All soft consistency scores between experts and GPT-4 exceed 0.75, with average differences below 1 point on each 5-point metric.The comparison covers Validness, Novelty, and Helpfulness (Table 12).Source: §A.8 · p. 14, Table 12 · p. 14

A.10 More Details About Past-Feedback Design

  • ChatGPT’s past-feedback often focused on inspiration relevance to the background, so the authors tested a heuristic favoring less-related inspirations, which improved MOOSE’s performance.The paper presents the framework as not limited to a particular LLM and suggests stronger models may generate better inspiration feedback.Source: §A.10 · p. 15

A.11 Qualitative Analysis on Hypotheses Generated From Different Methods

  • The expert analysis contrasts a conventional baseline hypothesis with more focused, nuanced proposals, including hypotheses about female CMOs’ empathy and business outcomes.The baseline received Validness 4, Novelty 1.5, and Helpfulness 2; the MOOSE hypothesis received 4.5, 4, and 4, respectively.Source: §A.11 · pp. 15–16

A.13 Essential Factors for Good Social Science (and Business) Hypotheses

  • Business PhD students favor counter-intuitive and novel hypotheses, estimating that about 20% of hypotheses in top business journals are counter-intuitive.A MOOSE example proposes that user-generated content can influence blind-box purchases among younger consumers, though its effect may vary across groups.Source: §A.13 · p. 17, §A.14 · pp. 18–19

A.14 An Example of Hypothesis Formulation via MOOSE

  • The example assumes positive rewards from surprise and omits negative user-generated content, other purchase drivers, and empirical evidence.Product quality, price, brand reputation, and peer influence may matter; disappointing experiences could discourage purchases.Source: §A.14 · p. 19
  • The blind-box hypothesis is judged novel and relatively clear, but its scope and specificity could be improved.It combines ideas about surprise, younger consumers, product differentiation, and user-generated content; suggested refinements include specifying the influence, audience, content types, and platforms.Source: §A.14 · pp. 19–20

A.15 Future Directions

  • Future work could extend hypothesis discovery beyond social science and business, improve MOOSE’s validity and novelty, and explore its use as a scientist’s AI copilot.Source: §A.15 · p. 20

A.16 Full Algorithm of the MOOSE Framework

  • The algorithm takes raw web corpus and related surveys as input, uses feedback iterations and hypothesis checks, and outputs a list of hypotheses.The listed checks assess clarity, reality, and novelty, while an inspiration-title finder uses the corpus and feedback.Source: §A.16 · p. 21
Loading 2309.02726v3…