PaperlayerView exact revision on arXiv

Source-linked AI summary

Literature Meets Data: A Synergistic Approach to Hypothesis Generation

Haokun Liu, Yangqiaoyu Zhou, Mingxuan Li, Chenfei Yuan, Chenhao Tan

arXiv:2410.17309v3cs.AIcs.CLcs.CYcs.LG

TL;DR

The paper asks whether literature-based and data-driven hypothesis generation can complement one another. It combines literature insights with data in an LLM-powered method, reporting stronger out-of-distribution accuracy than baselines and improved human accuracy on two tasks.

  • Problem

    Whether literature-based and data-driven hypothesis generation can complement each other remains an open question.

    Source: §1 · p. 1

  • Method

    The method combines literature insights with data for LLM-powered hypothesis generation, using literature-informed refinement alongside a data-driven hypothesis generator.

    Source: §1 · p. 1

  • Results

    8.97% over few-shot, 15.75% over literature-based alone, and 3.37% over data-driven alone in out-of-distribution accuracy; human accuracy improves by 7.44% on deception detection and 14.19% on AIGC detection.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Literature-based and data-driven hypotheses often contain novel information absent from the other set, and the paper concludes that integrating them can improve hypothesis generation and human decision-making.

    Source: §1 · p. 2, §6 · pp. 9–10

    The relatively small human-evaluation sample limits statistical power to distinguish methods such as HYPOGENIC and HYPOREFINE.

    Source: §7 · p. 10

Abstract

from arXiv · show

AI holds promise for transforming scientific processes, including hypothesis generation. Prior work on hypothesis generation can be broadly categorized into theory-driven and data-driven approaches. While both have proven effective in generating novel and plausible hypotheses, it remains an open question whether they can complement each other. To address this, we develop the first method that combines literature-based insights with data to perform LLM-powered hypothesis generation. We apply our method on five different datasets and demonstrate that integrating literature and data outperforms other baselines (8.97\% over few-shot, 15.75\% over literature-based alone, and 3.37\% over data-driven alone). Additionally, we conduct the first human evaluation to assess the utility of LLM-generated hypotheses in assisting human decision-making on two challenging tasks: deception detection and AI generated content detection. Our results show that human accuracy improves significantly by 7.44\% and 14.19\% on these tasks, respectively. These findings suggest that integrating literature-based and data-driven approaches provides a comprehensive and nuanced framework for hypothesis generation and could open new avenues for scientific inquiry.

1 Introduction

The paper combines literature-based and data-driven hypothesis generation, asking whether their strengths can complement each other. Across automatic and human evaluations, the integrated approach outperforms baselines on out-of-distribution accuracy and improves human accuracy on two tasks.

1 Introduction

  • Theory-driven methods draw on literature but require high-quality sources, adapt poorly to new data, and lack empirical support; data-driven methods adapt to data but may overfit their datasets.The paper frames integration as a response to these contrasting limitations.Source: §1 · p. 1
  • 8.97% over few-shot, 15.75% over literature-based alone, and 3.37% over data-driven alone in out-of-distribution accuracy show the integrated approach outperforms these baselines.Out-of-distribution accuracy is used as a measure of generalizability.Source: §1 · p. 2
  • Human accuracy improves by 7.44% on deception detection and 14.19% on AIGC detection, while literature-based and data-driven hypotheses often contribute novel information absent from the other set.This is presented as the first study assessing whether AI-generated hypotheses improve human decision-making.Source: §1 · p. 2
  • The paper proposes the first LLM-powered hypothesis-generation approach combining literature information and data, with a literature agent collaborating with HYPOGENIC to refine a shared hypothesis pool.It also proposes directly combining literature-based and data-driven hypotheses.Source: §1 · pp. 1–2

2 Methods

The method generates hypotheses for research questions using both relevant literature and observational data. It combines literature-only hypothesis generation, iterative data-driven hypothesis updating, and two integration strategies: unioning hypothesis banks or refining hypotheses with both sources.

2 Methods

  • The problem is to generate high-quality hypotheses for a research question q from relevant literature L and observational data D using an AI-powered algorithm.The paper considers research questions formulable as classification tasks.Source: §2 · p. 2

2.1 Literature-Based Hypothesis Generation

  • The literature-only pipeline selects relevant papers, converts and summarizes them, then prompts a language model to generate hypotheses from the summaries.The summaries focus on key findings to avoid context-length limits and distracting technical details.Source: §2.1 · p. 2

2.2 Data-Driven Hypothesis Generation

  • HYPOGENIC initializes hypotheses from examples, evaluates predictions on labeled data, and updates rewards; wrong examples can seed new hypotheses, and the final bank retains top hypotheses.The reward uses accuracy and an exploration term; selected hypotheses are evaluated as examples are visited.Source: §2.2 · p. 3

2.3 Integration of Literature-based and Data-driven Hypotheses

  • HYPOREFINE initializes hypotheses from paper summaries and examples, then iteratively applies data-driven and literature-based refinement before feeding the resulting bank back into HYPOGENIC.Refinement is applied to new hypotheses generated from the wrong-examples pool.Source: §2.3 · p. 3
  • The union strategy removes redundant hypotheses from literature-based and data-driven banks, then combines randomly selected literature hypotheses with top hypotheses from the other bank by training accuracy.Literature hypotheses may otherwise be undervalued or replaced by data-performing hypotheses that are not necessarily generalizable.Source: §2.3 · p. 3

3 Experiments

The experiments evaluate hypothesis utility and novelty through automatic and human studies across social-science tasks, comparing the proposed approach with prompting and hypothesis-generation baselines.

3.1 Evaluation Framework

  • Automatic evaluation uses hypotheses to guide inference on held-out OOD and IND sets, reports accuracy and F1 over five seeds, and tests cross-model transfer.For each example, the inference model selects relevant hypotheses before predicting.Source: §3.1 · p. 4
  • Human studies assess decision-making utility, perceived clarity, novelty, and plausibility, while separate pairwise annotations test whether literature- and data-driven hypotheses add distinct information.The utility study compares participants with and without three generated hypotheses; another study asks annotators to judge pairwise novelty.Source: §3.1 · p. 4

3.2 Tasks

  • Four tasks cover deception detection, AI-generated content detection, persuasive-argument prediction, and mental-stress detection.The tasks use datasets or constructed examples with in-distribution and out-of-distribution splits.Source: §3.2 · pp. 4–5

3.3 Implementation and Baselines

  • The method is presented as compatible with any LLM, with GPT-4O-MINI and LLAMA-3.1-70B-INSTRUCT used in the main paper.Baselines include zero- and few-shot prompting, direct zero-shot hypothesis generation, literature-driven methods, and HYPOGENIC; hypothesis banks are capped at 20.Source: §3.3 · p. 5

4 Results

The combined literature-and-data approach performs strongly in automatic evaluation, while human studies report improved decision accuracy and distinguish complementary hypothesis information from task-specific limits.

4.1 Automatic Evaluation

  • 3.37% average accuracy improvement over HYPOGENIC; Literature + Data is best across every task and model configuration on held-out OOD datasets.Table 1 reports OOD accuracy; its caption says bolded values significantly outperform few-shot inference.Source: §4.1 · pp. 5–6, Table 1 · p. 6
  • Literature refinement improves three tasks by 3.92% on average but drops accuracy by 13.64% on GPTGC and LLAMAGC, where interpretable literature cues are relatively sparse.The authors attribute the AIGC-task drop as likely related to limited literature insights on interpretable detection features.Source: §4.1 · p. 6
  • Case studies show HYPOREFINE can combine literature’s competitor-promotion insight with data’s lack-of-nuance cue into a more explanatory hypothesis.The example concerns balanced perspectives as indicators of truthful reviews.Source: §4.1 · p. 7
  • Literature + Data leads on most held-out IND cases, although HYPOGENIC tops selected Llama tasks; combining sources may slightly worsen IND performance.The reported HYPOGENIC-leading cases include GPTGC, LLAMAGC, and PERSUASIVE PAIRS.Source: §4.1 · p. 7
  • Cross-model transfer usually changes performance little, and prompt or hyperparameter changes also leave OOD results nearly stable.Performance drops by less than 5% in 11 of 20 transfer cases; prompt and hyperparameter changes reduce average performance by 0.20% and 0.07%, respectively.Source: §4.1 · pp. 7–8

4.2 Human Evaluation

  • Human accuracy rises 14.19% on AIGC detection and 7.44% on deception detection; participants use hypotheses for over 90% of decisions.The reported improvements are statistically significant, with p-values of 0.01 and 0.04, respectively; all participants found the hypotheses helpful.Source: §4.2 · p. 8
  • HYPOREFINE receives higher clarity and plausibility ratings than NOTEBOOKLM and HYPERWRITE, while NOTEBOOKLM’s slightly higher novelty rating is not statistically significant.The clarity and plausibility differences are reported as statistically significant.Source: §4.2 · p. 8
  • Annotators judge 84% of deception-detection pairs and 80% of AIGC-detection pairs as meaningfully novel across literature- and data-driven hypotheses.Novelty labels use majority votes from three annotators.Source: §4.2 · p. 8

5 Related Work

Related work spans literature-driven ideation, data-driven pattern discovery, and broader LLM research systems; this paper focuses on explanatory hypotheses grounded in observations and human decision-making evaluation.

Literature-based research idea generation

Data-driven hypothesis generation

  • Data-driven work discovers patterns through corpus differences, interpretable topics, program updates, or inductive reasoning; HYPOGENIC is selected as a compatible backbone (Zhong et al., 2023; Pham et al., 2024; Qiu et al., 2024).The paper says HYPOGENIC’s hypothesis updates integrate naturally into its refinement process.Source: §5 · p. 9

Automated scientific research with LLMs

  • Broader LLM research systems generate papers, implement research ideas, or support research tasks, whereas this work focuses on hypothesis generation to preserve human agency and oversight (Lu et al., 2024; Li et al., 2024).Source: §5 · p. 9

Evaluation of LLM-generated hypotheses

  • Prior evaluations examine novelty or expert ratings, but the authors identify human-subject validation of hypotheses’ decision-making utility as missing (Qi et al., 2023; Si et al., 2024).Source: §5 · p. 9

6 Conclusion

The paper proposes an approach that integrates literature and data for hypothesis generation and reports that it consistently outperforms baselines. Human evaluations also show improved decision-making on challenging tasks.

6 Conclusion

  • The paper proposes a novel approach that integrates literature and data to generate hypotheses.Source: §6 · p. 9
  • The integrated literature-and-data method consistently outperforms all baselines, including literature-based and data-driven approaches.The conclusion describes the evaluations as extensive and systematic.Source: §6 · p. 9
  • Human evaluations show that the generated hypotheses improve decision-making in challenging tasks.Source: §6 · p. 10

7 Limitations

This section discusses limitations in evaluation scope, corpus scale, tuning, human-subject evidence, hypothesis selection, and human–AI collaboration. It also documents prompt workflows and examples for hypothesis generation and refinement, including mental-stress detection and persuasive-argument prediction.

7 Limitations

  • The hypotheses were not evaluated on tasks requiring representations beyond natural language, including math problem solving and code generation.Automated evaluation used two recent models across datasets in various domains.Source: §7 · p. 10
  • The literature corpus was manually collected from Semantic Scholar or Google Scholar and capped at 10 papers, motivating future work on automatic, scalable retrieval.The authors report competent performance despite this limitation.Source: §7 · p. 10
  • Although initial hyperparameters performed satisfactorily across models and tasks, the study did not conduct an exhaustive search that might further improve performance.Source: §7 · p. 10
  • The human evaluation was a proof of concept with relatively few participants, limiting statistical power to distinguish methods such as HYPOGENIC and HYPOREFINE.The authors recommend larger-scale experiments in focused domains to validate hypotheses generated through human–AI collaboration.Source: §7 · p. 10
  • The authors manually selected three hypotheses for human-subject experiments using ablation-style study and subjective judgment, leaving the optimal human–AI collaboration regime for future exploration.Source: §7 · p. 10

A.2 AI-Generated Content (AIGC) Detection

  • AIGC prompts generate hypotheses from literature and refine them using observed text patterns, with refinements allowed to change a hypothesis’s key information or topic.The prompts then apply learned patterns to classify a text as AI- or human-written.Source: §A.2 · pp. 15–16

A.3 Mental Stress Detection

  • Stress-detection prompts require each hypothesis to pair a pattern indicating stress with its opposite, indicating no stress, and refine these paired hypotheses using posts or literature.The resulting patterns are applied to Reddit posts to predict whether the poster has stress.Source: §A.3 · pp. 16–17
  • A mental-stress detection example asks the model to classify a Reddit post as “has stress” or “no stress” and labels another example multiple-hypothesis-based inference.Source: §A.3 · p. 17

A.4 Persuasive Argument Prediction

  • Persuasive-argument prompts generate or refine hypotheses from paired arguments and observations or literature, seeking patterns that generalize to new instances.Source: §A.4 · pp. 17–18

B.1 Partitioning of IND and OOD datasets

The appendix details how the study partitions datasets and implements hypothesis generation, inference, and human-facing baselines. It also reports implementation settings, licensing, and estimated run costs.

B.1 Partitioning of IND and OOD datasets

  • IND/OOD partitions use review sources for deception detection, opposite generated-text datasets for GPTGC and LlamaGC, subreddit sources for mental-stress detection, and distinct corpora for persuasive-argument prediction.The deception-detection passage identifies 800 truthful and 800 deceptive IND reviews and describes the OOD set as reviews from four cities.Source: §B.1 · p. 19

B.2 Specificity Boost

  • The specificity booster expands short literature-only hypotheses with concrete examples for three tasks, but is omitted for Llama-3.1-70B-Instruct.The examples are based on the booster’s pre-training knowledge.Source: §B.2 · p. 19

B.3 Refinement and Union Implementation

  • HYPOREFINE initializes and updates hypotheses using paper summaries and examples, while the union procedure removes redundant hypotheses and combines banks with balanced prioritization.The union adds the top 10 from the data-driven bank, then selects literature-based hypotheses until the final bank reaches 20.Source: §B.3 · pp. 19–20

B.4 Multiple-Hypothesis Inference Implementation

  • Inference supplies a 20-hypothesis bank and a labeled-instance-free example to an LLM using chain-of-thought prompting, then reports held-out accuracy and macro-averaged F1.The prediction is computed from the hypothesis bank and instance.Source: §B.4 · p. 20

B.5 Technical Details of NotebookLM and HyperWrite

  • NotebookLM is evaluated as literature-based because it generates source-grounded responses from uploaded papers, while HyperWrite’s Hypothesis Maker uses a research question.The paper says HyperWrite has no publicly available technical report and generally uses pre-training knowledge and literature information.Source: §B.5 · p. 20

B.6 Hyperparameters

  • The study uses shared settings across tasks, models, and methods, including a 20-hypothesis bank, 200 training examples, six HYPOREFINE rounds, and five inference seeds.The passages also specify initialization and update parameters and common model-generation settings.Source: §B.6 · p. 20

B.7 Licensing Details; B.8 Estimated Cost

  • The paper reports compliance with dataset and model licenses and estimates that the Llama-70B pipeline takes 1.5 hours on four A100s, while the GPT-4-MINI run costs $0.6.The licensing passage identifies licenses for several datasets and models, including GPT-4-MINI’s proprietary status.Source: §B.7 · p. 20, §B.8 · p. 20

C.1 Decision-making Utility Study Details

The utility study compares unaided decisions with decisions made after reading three selected hypotheses, and separately gathers ratings and comparisons of generated hypotheses. The appendix also describes participant screening and consent procedures.

C.1 Decision-making Utility Study Details

  • Experimental-group participants read three hypotheses before predicting and could identify which they used; the study selected hypotheses whose removal most reduced inference performance.Participants also rated how helpful the hypotheses were.Source: §C.1 · p. 21
  • Human-accuracy calculations include only participants who finished on time and passed attention checks; 25 control and 22 experimental participants passed.The study recruited 30 participants per group; average completion time was around 25 minutes, with a 60-minute timeout.Source: §C.1 · p. 21

C.2 Likert Rating Survey Details

  • The deception-detection rating survey asks annotators to assess five representative hypotheses for clarity, novelty, and plausibility on five-scale metrics.Average human ratings are reported in Table 4.Source: §C.2 · p. 21

C.3 Novelty and Nuance Study Details

  • The novelty-and-nuance study compares literature-only and HYPOGENIC hypothesis banks for deception and AIGC detection, with 10 participants per task.All participants passed the attention-check question.Source: §C.3 · p. 21

C.4 IRB

  • The study proceeds only after participants consent; consent materials describe the study, incentives, risks, benefits, confidentiality, and contact information.The authors report receiving an IRB exemption.Source: §C.4 · p. 22

D Additional Experiments

This section introduces additional analyses of hypothesis-generation robustness.

D Additional Experiments

D.1 Llama-3.1-8B-Instruct Results

The Llama-3.1-8B-Instruct evaluation covers IND and OOD datasets across all tasks. Literature plus data outperforms other methods in most configurations and improves average accuracy over the listed baselines.

D.1 Llama-3.1-8B-Instruct Results

  • 15.27% average accuracy improvement over few-shot inference; literature plus data wins in 8 of 10 Llama-3.1-8B-Instruct configurations and beats the best literature-based method and HYPOGENIC by 13.04% and 4.88%, respectively.The evaluation covers IND and OOD datasets for all tasks.Source: §D.1 · p. 22

D.2 Robustness to Prompt Variations

Prompt variations leave average accuracy nearly unchanged across OOD and IND settings, while changes to Hmax also produce small average differences.

D.2 Robustness to Prompt Variations

  • Figure 2 presents the instruction page for the Likert-rating procedure.The figure is listed under the prompt-variation robustness subsection.Source: Figure 2 · p. 23
  • OOD accuracy decreases by 0.20% on average, while IND accuracy increases by 0.01% across the tested prompt variations.The variations modify the hypothesis-generation prompt, the inference prompt, or both.Source: §D.2 · pp. 22–23
  • For OOD prompt configurations, 71 of 90 have an accuracy drop under 5%, and 48 show an accuracy improvement.The three variations modify the hypothesis-generation prompt, the inference prompt, or both.Source: §D.2 · p. 23
  • For IND prompt configurations, 74 of 90 have a performance drop under 5%, and 50 show an improvement.The tested prompt variations modify the hypothesis-generation prompt, the inference prompt, or both.Source: §D.2 · pp. 22–23

D.3 Hyperparameter Search

  • The main experiments use HYPOGENIC's default hyperparameters, which the authors report work consistently across model and task configurations.The additional search varies Hmax among 10, 20, and 30.Source: §D.3 · p. 23
  • With Hmax set to 10, 20, or 30, OOD accuracy decreases by 0.07% on average; 31 of 60 configurations improve.Across those configurations, 51 of 60 have an accuracy drop under 5%.Source: §D.3 · p. 23
  • With Hmax set to 10, 20, or 30, IND accuracy decreases by 0.23% on average, with improvements in 26 of 60 cases.In 51 of 60 cases, performance drops by less than 5%.Source: §D.3 · p. 23

E Examples of Generated Hypotheses and Qualitative Analysis

The examples distinguish research ideas from hypotheses that explain phenomena, while the accompanying results describe performance across OOD and IND configurations.

E.2 Example Hypotheses

  • On held-out OOD datasets, Literature + data outperforms all other methods in every model and task configuration.Table 8 marks results significantly better than few-shot using a paired t-test over five random seeds.Source: Table 8 · p. 24

E.1 Comparing Hypotheses from SciMON and Ours

  • SciMON proposes a research project method, whereas HYPOREFINE offers a possible explanation for deception detection, relevant to Li et al. (2014).The example contrasts SciMON's idea with the existing deception-detection finding from Li et al. (2014) (Wang et al., 2024).Source: §E.1 · pp. 23–24, Table 15 · p. 35

E.2 Example Hypotheses

  • On DECEPTIVE REVIEWS, invalid or irrelevant hypotheses from NOTEBOOKLM and HYPERWRITE can degrade inference performance.Table 17 presents examples of these hypotheses.Source: §E.2 · p. 24, Table 17 · p. 37
  • On held-out IND datasets, Literature + data leads in 7 of 10 configurations, while HYPOGENIC leads on three LLAMA-70B-I tasks.Those tasks are GPTGC, LLAMAGC, and PERSUASIVE PAIRS; the authors suggest literature may be unhelpful for their IND data but useful for the tasks generally.Source: Table 9 · p. 25
  • Figures 3–5 show Likert-rating and novelty-check instructions or annotation pages.The figures include the novelty-check instruction page and its annotation page.Source: Figure 3 · p. 26, Figure 4 · p. 27, Figure 5 · p. 28
  • Figures 6–9 cover prediction-task instructions and annotation pages for conditions without hypotheses and with hypotheses as a guide.The paired conditions allow readers to compare the prediction task setups.Source: Figure 6 · p. 28, Figure 7 · p. 29, Figure 8 · p. 30
  • With Llama-3.1-8B-Instruct, Literature + data outperforms other methods in 4 of 5 configurations for both OOD and IND datasets.The authors describe this as evidence of effectiveness with a smaller model.Source: Table 10 · p. 32

Related papers

Loading 2410.17309v3…