PaperlayerView exact revision on arXiv

Source-linked AI summary

Hypothesis Generation with Large Language Models

Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, Chenhao Tan

arXiv:2404.04326v3cs.AIcs.CLcs.CYcs.LG

TL;DR

The paper asks how LLMs can generate high-quality hypotheses from labeled examples despite long-context and quality-assessment challenges. HypoGeniC iteratively generates hypotheses and uses them for inference, outperforming few-shot prompting across four tasks while producing interpretable hypotheses and new insights.

  • Problem

    LLMs may not effectively use examples in one long prompt, and hypothesis generation needs measures of quality to filter and improve hypotheses.

    Source: §1 · p. 1

  • Method

    HypoGeniC iteratively generates hypotheses from labeled examples and uses the resulting hypotheses to make inferences.

    Source: §1 · p. 1, §2 · p. 2, §6 · p. 8

  • Results

    31.7% in SHOE SALES: generated hypotheses consistently outperform few-shot in-context learning across all four tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The generated hypotheses are human-interpretable and uncover new insights in the studied real-world tasks.

    Source: §1 · p. 2, §6 · p. 8

    The authors warn that data may reinforce stereotypes or expose personal information, recommending human-AI collaboration and oversight.

    Source: §7 · p. 9

Abstract

from arXiv · show

Effective generation of novel hypotheses is instrumental to scientific progress. So far, researchers have been the main powerhouse behind hypothesis generation by painstaking data analysis and thinking (also known as the Eureka moment). In this paper, we examine the potential of large language models (LLMs) to generate hypotheses. We focus on hypothesis generation based on data (i.e., labeled examples). To enable LLMs to handle arbitrarily long contexts, we generate initial hypotheses from a small number of examples and then update them iteratively to improve the quality of hypotheses. Inspired by multi-armed bandits, we design a reward function to inform the exploitation-exploration tradeoff in the update process. Our algorithm is able to generate hypotheses that enable much better predictive performance than few-shot prompting in classification tasks, improving accuracy by 31.7% on a synthetic dataset and by 13.9%, 3.3% and, 24.9% on three real-world datasets. We also outperform supervised learning by 12.8% and 11.2% on two challenging real-world datasets. Furthermore, we find that the generated hypotheses not only corroborate human-verified theories but also uncover new insights for the tasks.

1 Introduction

The paper addresses how LLMs can generate high-quality hypotheses from labeled examples despite long contexts and the difficulty of assessing hypothesis quality. It introduces HypoGeniC, an iterative generation framework, and reports stronger predictive performance than few-shot baselines across four tasks, alongside cross-model results and new task insights.

1 Introduction

  • The paper studies data-based hypothesis generation because LLMs may not effectively use input examples in one long prompt, and generation needs a way to assess and improve hypothesis quality.Training accuracy guides generation, while held-out performance evaluates the final hypotheses.Source: §1 · p. 1
  • HypoGeniC starts with hypotheses from a small number of examples, then uses a reward function and wrong-example bank to guide iterative generation.The method evaluates top hypotheses on each training example and uses examples missed by enough hypotheses to generate new ones.Source: §1 · p. 1, Figure 1 · p. 2
  • The framework produces interpretable hypothesis-based classifiers evaluated on one synthetic task and three real-world tasks involving deception detection and message popularity prediction.The synthetic task has a known valid hypothesis; the real-world tasks are DECEPTIVE REVIEWS, HEADLINE POPULARITY, and TWEET POPULARITY.Source: §1 · p. 1
  • 31.7% in SHOE SALES, 13.9% in DECEPTIVE REVIEWS, 3.3% in HEADLINE POPULARITY, and 24.9% in TWEET POPULARITY: generated hypotheses outperform few-shot in-context learning across all four tasks.Their predictive performance matches or exceeds oracle supervised learning with RoBERTa and Llama-2-7B except in DECEPTIVE REVIEWS.Source: §1 · p. 2
  • Generated hypotheses can support inference across different LLMs and an out-of-distribution DECEPTIVE REVIEWS dataset, and the authors report corroborating prior theories while uncovering new task insights.They describe using hypotheses generated by GPT-3.5-turbo for inference by Mixtral and report outperforming oracle fine-tuned RoBERTa on the out-of-distribution dataset.Source: §1 · p. 2

2 Method

The method learns data-derived hypotheses, updates them to address gaps in their coverage, and uses the resulting hypotheses to infer labels.

2 Method

  • Given labeled examples, the goal is to learn hypotheses describing relationships between examples and labels, then use them to make inferences.The LLM summarizes demonstration examples into hypotheses before inference.Source: §2 · p. 2

2.1 Hypothesis Generation

  • The update stage selects high-reward hypotheses for each training example and uses accumulated wrong examples to generate hypotheses that address gaps in the current bank.New hypotheses are incorporated according to their rewards.Source: §2.1 · p. 3
  • The reward balances each hypothesis’s accuracy on evaluated examples against an exploration term that favors hypotheses selected less often.The exploration term depends on the number of selections and training examples visited; α controls its strength.Source: §2.1 · p. 3

Algorithm 1 HypoGeniC

  • Algorithm 1 initializes a hypothesis bank, evaluates its top-k hypotheses on training examples, accumulates wrong examples, and retains top-reward hypotheses after generating replacements.The wrong-example pool triggers new hypothesis generation when it reaches wmax.Source: Algorithm 1 HypoGeniC · p. 3, §2.1 · p. 3

2.2 Hypothesis-based Inference

  • Inference strategies use the most accurate hypothesis, a filtered weighted vote, or adaptive hypothesis selection in one or two prompting steps.The weighted vote uses hypotheses’ training accuracy; adaptive strategies select hypotheses relevant to each test example.Source: §2.2 · pp. 3–4

3 Experiment Setup

The experiments assess hypothesis generation on four classification tasks, comparing HypoGeniC with prompting, no-update, and supervised-learning baselines.

3.1 Tasks and Datasets

  • Four tasks comprise a controlled six-class shoe-color task, deceptive-review detection, and headline- and tweet-popularity comparisons, selected for complex patterns not readily known to LLMs.The shoe task has one valid hypothesis; the real-world tasks address challenging social-science problems.Source: §3.1 · p. 4

3.2 Baselines, Oracles, and Evaluation Metrics

  • Comparisons include zero- and few-shot prompting, initialized hypotheses without updates, and fine-tuned RoBERTa and Llama-2-7B; accuracy is measured across 10–200 training examples and two bank sizes.The experiments use Mixtral, GPT-3.5-turbo, and Claude-2.1, with 200 training and 300 test examples sampled per dataset.Source: §3.2 · p. 4

3 Experiment Setup

  • The setup evaluates methods on classification tasks with ground-truth labels, using accuracy as the evaluation metric.The experiments vary training-set size and hypothesis-bank size.Source: §3.2 · p. 4

4 Results

The results assess predictive performance, cross-model and out-of-distribution generalization, and whether generated hypotheses corroborate prior findings or reveal new insights. The authors report gains over few-shot inference and, on some real-world datasets, supervised models.

4.1 Performance on Heldout Test Sets

  • HypoGeniC improves over few-shot inference by 31.7% on SHOE SALES, 13.9% on DECEPTIVE REVIEWS, 3.3% on HEADLINE POPULARITY, and 24.9% on TWEET POPULARITY.The authors note that TWEET POPULARITY results are inflated by safety mode for some baselines; the GPT-3.5-turbo few-shot baseline is 1% better on that task.Source: §4.1 · p. 5
  • With 200 examples, HypoGeniC beats RoBERTa by 12.8% on HEADLINE POPULARITY and 11.2% on TWEET POPULARITY, and matches or exceeds fine-tuned models on most datasets.With fine-tuned models given 1,000 examples, HypoGeniC’s best result still exceeds RoBERTa by 3.7% and 0.7%, and Llama-2-7B by 3.7% and 11.4%, on HEADLINE POPULARITY and TWEET POPULARITY, respectively.Source: §4.1 · p. 5
  • Updating hypotheses improves average performance over no updates by 0.7% on SHOE SALES, 5.8% on DECEPTIVE REVIEWS, 8.1% on HEADLINE POPULARITY, and 7% on TWEET POPULARITY.The authors also report that updates can help when examples exceed an LLM’s context window.Source: §4.1 · p. 5
  • Single-step adaptive inference is most effective on DECEPTIVE REVIEWS; best-accuracy suffices for SHOE SALES and HEADLINE POPULARITY, while filter and weighted vote work best for TWEET POPULARITY.Performance generally improves with more examples and a larger hypothesis pool, though real-world datasets sometimes peak before 200 examples; a pool of 20 outperforms one of 3.Source: §4.1 · pp. 5–6

4.2 Generalization of the Generated Hypotheses

  • Hypotheses generalize across inference models: Claude-2.1 or Mixtral often improve inference with GPT-3.5-turbo-generated hypotheses, while switching between Claude-2.1 and Mixtral changes results only modestly.On DECEPTIVE REVIEWS and TWEET POPULARITY, switching from Claude-2.1 to Mixtral reduces performance by 2.3% and 2.7%, respectively.Source: §4.2 · p. 6
  • On out-of-distribution deceptive reviews from four cities and different websites, HypoGeniC beats few-shot learning by an average of 19.1% and improves over its original-dataset accuracy by 3.3%.On the shifted dataset, RoBERTa drops by 11% and Llama-2-7B by 10%; HypoGeniC with Claude-2.1 exceeds RoBERTa by 1.7%.Source: §4.2 · p. 7

4.3 Qualitative Analysis

  • Generated hypotheses recapitulate reported features of deceptive reviews, popular tweets, and clickable headlines, while also identifying new task insights.Examples include truthful hotel reviews stating the reviewer’s purpose, relatable headlines receiving more clicks, and tweets mentioning influential people or organizations being retweeted more.Source: §4.3 · p. 7
  • The analysis highlights contextual ambiguity in the word “future” and finds lower-accuracy hypotheses linking straightforward or clear headline writing to higher click-through rates, consistent with a prior negative reading-ease finding.The authors contrast contextual interpretation with feature engineering, which they say is limited by ambiguity around individual tokens.Source: §4.3 · pp. 7–8

5 Additional Related Work

The related work spans LLM-driven concept and hypothesis generation, reasoning, and applications in social science. The paper distinguishes its data-to-label hypothesis generation and verification focus from these neighboring approaches.

Concept/pattern discovery

  • Prior work explores LLM concept induction, open-ended exploration, program or topic discovery, and hypotheses from literature or web corpora; this paper targets hypotheses linking inputs to labels in challenging real-world tasks.The authors distinguish their UCB-style reward for proposing novel algorithms from approaches requiring existing fact-rule pairs or human-annotated hypotheses.Source: §5 · p. 8

Reasoning with LLMs

  • Unlike chain-of-thought reasoning, hypothesis generation has no predefined reasoning structure and requires proposing and verifying hypotheses to discover unknown knowledge.The paper also relates hypothesis-based classifiers to reasoning with LLMs.Source: §5 · p. 8

LLMs for (social) sciences

  • The experiments demonstrate LLM hypothesis generation’s potential for social science research, and the authors suggest extending the approach to natural sciences.Source: §5 · p. 8

6 Conclusion & Further Discussion

The conclusion presents HypoGeniC as an automated, data-driven approach for generating interpretable hypotheses and discovering unknown knowledge. It attributes the approach’s success to exploration that selects useful combinations of concepts and points to broader scientific applications.

6 Conclusion & Further Discussion

  • HypoGeniC generates human-interpretable hypotheses, achieves better predictive performance than competitive baselines and some oracles, and generalizes across models and datasets.The authors particularly note new insights in widely studied social-science tasks.Source: §6 · p. 8
  • The authors attribute HypoGeniC’s success to LLMs combining potentially relevant concepts, with exploration–exploitation identifying valuable hypotheses rather than relying on remembered correct ones.Source: §6 · p. 8
  • The authors propose exploring hypothesis generation that uses additional modalities or combines existing literature with past observations.Source: §6 · p. 8

7 Limitations

This section discusses the evaluation’s scope and uncertainties in inference, tuning, cost, and safe deployment, and provides prompt implementations for headline-click and tweet-retweet prediction. It also notes that hypotheses may reveal patterns even when accuracy gains are not significant.

7 Limitations

  • The evaluation focuses on social-science tasks because math and physics hypotheses are difficult to represent in natural language and often require symbolic parsers.Extending the framework to natural-science tasks is left for future work.Source: §7 · p. 9
  • Hypotheses may reveal previously unseen patterns even when accuracy improvement is not significant, though imperfect LLM reasoning may fail to reflect their quality.The authors refer to § 4.3 for the new patterns and caution that hypothesis-based inference may not accurately assess hypothesis quality.Source: §7 · p. 9
  • Zero/few-shot results suggest models do not solve the tasks by memorizing data, and generated hypotheses were not found in the literature the authors manually searched.The authors say this suggests even experienced researchers struggle to find the generated hypotheses.Source: §7 · p. 9
  • The authors did not extensively tune generation: they left k unchanged, tested hypothesis-bank sizes of 3 and 20, and tested wmax only at 10.They note that the ideal bank size needs further investigation and that more thorough hyperparameter search could improve performance.Source: §7 · p. 9
  • Inference latency is high for multi-prompt methods, while single-step adaptive inference and best-accuracy-hypothesis inference are described as efficient.Across 76 experiments, GPT-3.5-turbo averaged $2.05 and 1.5 hours per experiment; open models alleviate cost concerns, and the authors say the processes remain relatively cheap compared with human effort.Source: §7 · p. 9
  • Using HypoGeniC without proper oversight could reinforce stereotypes or biases or expose personal information, so the authors recommend human-AI collaboration.The authors say there is little guard against confirming biases in data and no guarantee that generated hypotheses will not reveal personal information.Source: §7 · p. 9

A.3 Headlines With More Clicks

  • For headline clicks, the prompt selects the pattern whose examples most closely match a new headline pair, then applies it to predict which headline gets more clicks.The adaptive prompt contains hypotheses and the examples each correctly predicted during generation.Source: §A.3 · p. 14

A.4 Retweeted More

  • For retweet prediction, the generation prompt asks for general hypotheses based on wording differences between tweets about the same content from the same user.It requests multiple hypotheses in a specified numbered format.Source: §A.4 · p. 14
  • Retweet inference prompts either apply a learned pattern to a tweet pair or select a pattern by matching its examples to the pair before predicting.The examples also include zero/few-shot inference and adaptive hypothesis selection using hypotheses with examples correctly predicted during generation.Source: §A.4 · pp. 14–15

B Implementation and Setup Details

HypoGeniC samples examples to initialize rewards fairly across hypotheses generated at different training stages, and adjusts how often it updates the hypothesis pool over time.

B.1 HypoGeniC implementation

  • Reward initialization can use both wrong-bank examples and initial examples to reduce accuracy bias for hypotheses generated late in training.Using more examples makes reward accuracy closer to its true value, enabling fairer comparisons with earlier hypotheses.Source: §B.1 · p. 15
  • Sampling from initial examples supplements the wrong-example bank when initializing rewards for newly generated hypotheses.This addresses the likely bias in accuracy estimates for hypotheses generated near the end of training in the low-data regime.Source: §B.1 · p. 15
  • New examples enter the wrong-example bank when more than whyp hypotheses predict them incorrectly.The implementation increases whyp linearly with training time, producing more frequent updates early and less frequent updates later.Source: §B.1 · p. 15

B.2 Inference method implementations

Inference either filters and combines highly rewarded relevant hypotheses or prunes overlapping hypotheses before selection. The implementation also specifies bank-size-dependent settings and reports licensing details.

B.2 Inference method implementations

  • Filter and weighted vote checks top-ranked hypotheses for relevance, then predicts with relevant hypotheses and selects the label with the greatest summed accuracy.Hypotheses are ranked by reward before relevance is checked.Source: §B.2 · p. 15
  • Adaptive inference prunes overlapping hypotheses by comparing their correct-prediction patterns and selecting a high-accuracy subset below a pairwise similarity threshold.The method encodes correctly predicted examples, computes pairwise cosine similarities, and uses a linear program to maximize selected-hypothesis accuracy subject to threshold γ.Source: §B.2 · pp. 15–16
  • After pruning, single-step adaptive inference selects a hypothesis and predicts in one prompt, while two-step inference uses separate selection and prediction prompts.Both strategies prompt the LLM to select one hypothesis for its final prediction.Source: §B.2 · p. 16

B.3 Hyperparameters

  • Across experiments, the reward coefficient was α = 0.5, wmax = 10, and num_init = 10, with hypothesis-bank sizes of H = 3 or H = 20.The remaining hyperparameters differed between the two bank-size settings.Source: §B.3 · p. 16
  • For H = 3, the method used k = 2 and generated one hypothesis per update; for H = 20, it used k = 10 and generated five per update.Inference used all three hypotheses in the smaller bank and the top five in the larger bank; adaptive inference used γ = 0.3 and γ = 0.7, respectively.Source: §B.3 · p. 16

B.4 Licensing Details

  • Dataset and model licenses vary, and the authors report compliance with the licenses of all datasets and models used.The datasets include research-use permissions and Creative Commons licenses; the models include proprietary, Apache, research-use, and model-specific licensing.Source: §B.4 · p. 16

C.1 HypoGeniC Performance across inference strategies

The strongest inference strategy varies by dataset: single-step adaptive inference leads on DECEPTIVE REVIEWS, while other tasks favor different policies. The analysis links those differences to how hypotheses and examples are used during prediction.

C.1 HypoGeniC Performance across inference strategies

  • For SHOE SALES, single-step adaptive inference drops in accuracy for all models, and the authors suggest the long prompt may make selecting the correct hypotheses difficult.Claude-2.1 also drops with filter and weighted vote, which the authors attribute to inaccurate relevant patterns outweighing the best hypothesis.Source: §C.1 · p. 16
  • Single-step adaptive inference is most effective on DECEPTIVE REVIEWS, where the prompt explicitly asks whether a review is deceptive and may help match test reviews to examples.The authors suggest splitting the prompt may hurt performance.Source: §C.1 · p. 16
  • HEADLINE POPULARITY is the most challenging dataset because the evaluation uses headlines without the paired images that contribute to click behavior.Despite that constraint, the highest reported accuracy is 63.7%, with the best-accuracy hypothesis strategy performing best.Source: §C.1 · p. 16
  • Filter and weighted vote is best on TWEET POPULARITY, while best-accuracy hypothesis is sufficient for good performance on SHOE SALES and HEADLINE POPULARITY.The authors suggest combining hypotheses can capture multiple variables in TWEET POPULARITY.Source: §C.1 · p. 17, Figure 3 · p. 17
  • Statistically significant improvements over few-shot baselines are reported for DECEPTIVE REVIEWS and TWEET POPULARITY with Claude-2.1 and Mixtral.The passage reports significance using 95% confidence intervals under a binomial-distribution assumption.Source: §C.1 · p. 17

C.2 HypoGeniC Performance across training examples

Performance across training-example counts depends on the model, dataset, and hypothesis-bank size. Updates often outperform no updates, but more examples do not consistently improve accuracy.

C.2 HypoGeniC Performance across training examples

  • With Claude-2.1 and a hypothesis bank of 20, HypoGeniC outperforms No Updates across all training-example counts and four datasets.For HEADLINE POPULARITY, it needs 200 examples to outperform zero-shot and few-shot learning; more examples do not necessarily increase accuracy.Source: §C.2 · p. 17
  • GPT-3.5-turbo results are mixed against few-shot learning, and few-shot surpasses HypoGeniC on TWEET POPULARITY, although HypoGeniC exceeds zero-shot there.HypoGeniC generally surpasses No Updates; that baseline fails when GPT-3.5-turbo cannot accept the training examples within its context window.Source: §C.2 · pp. 17–18
  • Table 5 presents 95% confidence intervals for few-shot results and HypoGeniC’s best results.The supplied caption does not specify the table’s numerical comparisons.Source: Table 5 · p. 18
  • With Mixtral, HypoGeniC outperforms zero-shot and few-shot strategies on all datasets except SHOE SALES, where it needs 200 examples to beat few-shot.For TWEET POPULARITY, No Updates surpasses HypoGeniC with bank size 3 after 200 examples; the authors suggest three hypotheses may be too few to describe the dataset.Source: §C.2 · p. 18

C.3 Full OOD results

On the out-of-distribution deceptive-reviews dataset, HypoGeniC outperforms zero-shot and few-shot learning across models and inference policies. Robustness varies by policy, and the authors describe a possible source of single-step performance drops.

C.3 Full OOD results

  • HypoGeniC outperforms zero-shot and few-shot learning across all models and inference policies on the OOD deceptive-reviews dataset.Best-accuracy and two-step adaptive inference show average increases of 3.7% and 3.6%, respectively.Source: §C.3 · p. 18
  • Although filter and weighted vote may appear to have mixed performance, the passage says the strategy remains robust; Mixtral’s accuracy drop under it is minimal at 1%.Source: §C.3 · p. 18
  • Single-step adaptive inference drops on OOD reviews, which the authors suggest may occur because training examples remain in the context and influence predictions.For two-step adaptive inference, a dedicated hypothesis-selection prompt leaves application of the hypothesis unaffected by those training samples.Source: §C.3 · p. 18

D Qualitative Analysis on Generated Hypotheses

The qualitative analysis finds that generated hypotheses on the real-world datasets include both ideas supported by existing findings and novel ideas. The authors suggest these hypotheses may guide future research.

D Qualitative Analysis on Generated Hypotheses

Loading 2404.04326v3…