PaperlayerView exact revision on arXiv

Source-linked AI summary

ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition

Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, Dongzhan Zhou

arXiv:2503.21248v3cs.CLcs.AIcs.CE

TL;DR

LLMs’ ability to discover high-quality scientific hypotheses has lacked a dedicated benchmark. This paper introduces ResearchBench, which decomposes discovery into three sufficient subtasks and builds a scalable, contamination-resistant benchmark from papers across 12 disciplines. LLMs perform especially well on out-of-distribution inspiration retrieval, while the benchmark also evaluates composition and ranking.

  • Problem

    The ability of LLMs to discover high-quality research hypotheses remains insufficiently evaluated because dedicated benchmarks are lacking.

  • Method

    ResearchBench evaluates inspiration retrieval, hypothesis composition, and hypothesis ranking using an automated framework that extracts research components from papers across 12 disciplines.

  • Results

    LLMs excel at out-of-distribution inspiration retrieval across disciplines and also perform well on hypothesis composition and ranking.

  • Takeaways & Limitations

    The findings position LLMs as potential research hypothesis mines capable of surfacing novel knowledge associations at scale.

  • Takeaways & Limitations

    ResearchBench currently covers only 12 disciplines selected primarily according to data and resource availability, and it does not target fine-grained hypothesis discovery.

Abstract

from arXiv · show

Large language models (LLMs) have shown potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined due to the lack of a dedicated benchmark. To address this gap, we introduce the first large-scale benchmark for evaluating LLMs on a sufficient set of scientific discovery sub-tasks-inspiration retrieval, hypothesis composition, and hypothesis ranking-where sufficient means that perfectly solving these sub-tasks perfectly solves the overall discovery task. We develop an automated LLM-based framework that extracts critical components-research questions, background surveys, inspirations, and hypotheses-from papers across 12 disciplines, with expert validation confirming its accuracy. To prevent data contamination, we focus exclusively on publications from 2024 onward, ensuring minimal overlap with LLM pretraining data; our automated framework further enables automatic extraction of even more recent papers as LLM pretraining cutoffs advance, supporting scalable and contamination-free automatic renewal of this discovery benchmark. Our evaluation shows that, across disciplines, LLMs excel at inspiration retrieval-an out-of-distribution task-suggesting their ability to surface novel knowledge associations.

1 Introduction

ResearchBench addresses the lack of a dedicated evaluation of LLMs for scientific hypothesis discovery by decomposing discovery into inspiration retrieval, hypothesis composition, and hypothesis ranking. It builds a scalable benchmark from recent scientific papers and finds strong LLM performance in inspiration retrieval, alongside performance in composition and ranking.

  • Motivation: LLM scientific hypothesis discovery is difficult to evaluate because existing benchmarks do not measure this capability directly.The benchmark targets the task of helping scientists discover valid new hypotheses from a research question and background survey.
  • Task decomposition: The proposed decomposition treats inspiration retrieval, hypothesis composition, and hypothesis ranking as a sufficient set of discovery subtasks.Under the stated assumption, perfectly solving these subtasks perfectly solves the overall discovery task.
  • Benchmark construction: ResearchBench covers 12 disciplines and selects papers published from 2024 onward to reduce contamination from pretraining data.The benchmark is intended to renew automatically with newer papers as pretraining cutoffs advance.
  • Benchmark construction: An automated LLM-based framework extracts research questions, background surveys, inspirations, and hypotheses from scientific papers.The framework was applied to 1386 papers and its decomposition accuracy was checked by five experts.
  • Findings: 45.7% is the probability that a ground-truth inspiration appears among GPT-4o’s top 4% of candidates.Across disciplines, LLMs perform well on inspiration retrieval despite challenging negative examples, and also perform well on composition and ranking.
  • Findings: The study presents LLMs as potential research hypothesis mines based on their performance across the three discovery subtasks.The inspiration retrieval task is characterized as out-of-distribution because useful inspirations are not known in advance to be related to the research question.

2 Related Work

Prior work supports decomposing scientific discovery into inspiration retrieval, hypothesis composition, and hypothesis ranking, but existing benchmarks cover only parts of this process or narrower domains. ResearchBench is positioned as a broader benchmark spanning the full decomposition across multiple disciplines.

  • Foundations: Prior studies describe hypotheses as compositions of research backgrounds and several inspirations, motivating a decomposition of scientific discovery.The decomposition is grounded in discussions with domain experts and cognitive-science findings about combining seemingly unrelated knowledge.
  • Foundations: The proposed sufficient decomposition consists of retrieving inspirations, composing hypotheses from background information and inspirations, and ranking the resulting hypotheses.This formulation treats the three subtasks as sufficient for solving the overall discovery task.
  • Benchmark gap: Existing benchmarks assess general intelligence or narrower discovery tasks without evaluating the full set of scientific discovery subtasks.IdeaBench focuses on biomedical hypothesis generation, while DiscoveryBench and ScienceAgentBench extract other discovery-relevant tasks from limited paper collections.
  • Benchmark gap: ResearchBench differs by covering the complete decomposition across 12 disciplines with an LLM-based agentic extraction framework.Its construction addresses limitations in domain coverage, inspiration retrieval, integration, and extraction methodology identified in prior benchmarks.

3 Benchmark Construction

The benchmark decomposes scientific discovery into inspiration retrieval, hypothesis composition, and hypothesis ranking, then builds a 12-discipline dataset with automated extraction, contamination controls, and expert validation. Its evaluation design uses challenging negative inspirations and pairwise or top-ranked selection to assess retrieval and discovery-related capabilities.

  • 3.1 Theoretical Foundation: Scientific discovery is decomposed into inspiration retrieval, hypothesis composition, and hypothesis ranking, with perfect performance on all three defined as sufficient for overall discovery.The decomposition assumes that hypotheses can originate from a research background and several inspirations.
  • 3.2 Benchmark Scope: The benchmark spans 12 disciplines and collects papers from top-ranked venues to evaluate the decomposed scientific-discovery tasks.The construction includes research backgrounds, inspirations, and hypotheses extracted from each paper.
  • 3.2 LLM-Based Agentic Framework: An LLM-based agentic framework automatically extracts research questions, background surveys, inspirations, and hypotheses from papers.Research questions, background surveys, and hypotheses use carefully designed prompts and iterative self-refinement; inspiration extraction first identifies referenced-paper candidates.
  • 3.2 LLM-Based Agentic Framework: Papers published from 2024 onward are used to minimize overlap with LLM pretraining data and support future benchmark renewal.The framework can be applied to newer papers as pretraining cutoffs advance.
  • 3.3 Negative Inspiration Selection: Negative inspirations are organized into three distance levels, combining citation- or semantic-adjacent papers, same-discipline papers, and cross-discipline papers.Each benchmark paper receives 25 randomly selected negatives from each level; close negatives make retrieval evaluation more challenging and allow preference by distance to be analyzed.
  • 3.4 Expert Evaluation: Expert review of 62 sampled papers found 91.9% decomposition accuracy for major issues and 82.3% when minor issues were included.Five experts evaluated extracted research questions, background surveys, inspirations, and hypotheses; the review identified five major and six minor issues.

4 Experiments

The benchmark evaluates LLMs on inspiration retrieval, hypothesis composition, and hypothesis ranking using twelve models and paper-derived inputs. LLMs retrieve inspirations relatively well, while composition remains challenging and ranking benefits from larger scale and improved training.

  • Experimental Setup: The experiments evaluate twelve representative LLMs across inspiration retrieval, hypothesis composition, and hypothesis ranking.The benchmark uses paper-derived research questions, backgrounds, inspirations, and hypotheses, with concrete task examples provided in Appendix A.8.
  • 4.1 Inspiration Retrieval: Retrieval narrows 75 candidates to 3 through two selection rounds, retaining 4% of the original set for final evaluation.Candidates are partitioned into five groups of 15, reduced to 15 and then 3 overall.
  • 4.1 Inspiration Retrieval: At 4% retention, GPT-4o achieves 45.65% retrieval accuracy, while most models identify about 80% of ground-truth inspirations at 20% retention.The final round selects only three papers, yet accuracy remains above 40%.
  • 4.1 Inspiration Retrieval: Negative inspirations closer to the research question are selected more often, consistent with their greater statistical relevance and frequent pretraining co-occurrence.This pattern holds across distance levels and selection percentages.
  • 4.2 Hypothesis Composition: Hypothesis composition combines research background and inspirations through mutation and recombination, but none of the evaluated models fully solves the task.Performance is measured by normalized average scores on a 0–5 Likert scale against key points in ground-truth hypotheses.
  • 4.3 Hypothesis Ranking: Hypothesis ranking improves with larger models and better pretraining strategies, unlike inspiration retrieval, which shows weaker gains at larger scale.Ranking accuracy is measured over 15 pairwise comparisons, with reversed presentation order used to reduce position bias.
  • 4.3 Hypothesis Ranking: Position bias remains substantial: Llama-3.1-8B produces self-contradictory rankings in 91.67% of comparisons, versus 19.17% for Claude 3.5 Sonnet.These self-contradictory outcomes likely help explain why many models achieve approximately 50% ranking accuracy.

5 Analysis

The analysis presents LLMs as capable of retrieving latent knowledge associations, composing hypotheses from correct inspirations, and improving ranking with scale and training. It also identifies distinct limitations in cross-domain retrieval, novelty of composition, and positional bias in ranking.

  • 5.1 LLMs as Research Hypothesis Mines: LLMs capture latent knowledge associations that support accurate inspiration retrieval, including associations beyond established links.The authors frame this capability as enabling LLMs to function as research hypothesis mines.
  • 5.1 LLMs as Research Hypothesis Mines: Given correct inspirations, LLMs compose hypotheses that reflect key elements of the original innovations.The three evaluated subtasks are presented as sufficient for scientific discovery under the paper’s decomposition.
  • 5.1 LLMs as Research Hypothesis Mines: Larger scale and better training improve hypothesis ranking without a clear limit, whereas retrieval and composition expose different constraints.The analysis distinguishes improvement in ranking from the retrieval bottleneck and composition difficulty reported elsewhere.
  • 5.2 Failure Cases: Inspiration-retrieval errors chiefly reflect title or abstract overlap judgments, missed cross-disciplinary terminology, and misunderstandings of the research question.Among 100 sampled retrieval errors, 56% involved overlap-based relevance judgments and 23% involved missed papers from other disciplines.
  • 5.2 Failure Cases: Ranking errors are dominated by comparison-order processing, while subtler cases involve distinguishing between two reasonable hypotheses.83% of sampled ranking errors arose from order processing, and 11% involved subtle distinctions between reasonable hypotheses.
  • 5.3 Insights & Challenges: Cross-domain inspiration retrieval is constrained by training-data distributions with dense intra-domain co-occurrences but sparse interdomain links.The stated bottleneck lies in training-data distribution rather than model reasoning, limiting retrieval of weakly related inspirations.
  • 5.3 Insights & Challenges: Autoregressive prediction constrains hypothesis ranking by creating positional dependencies that bias models toward the first input.The resulting bias produces systematic errors in pairwise comparisons.

6 Conclusion

ResearchBench is a large-scale benchmark for LLM-driven scientific discovery across inspiration retrieval, hypothesis composition, and hypothesis ranking. Its results show strong retrieval but only moderate composition and ranking, indicating useful associations without deeper integrative reasoning.

  • ResearchBench benchmarks LLM-driven scientific discovery across inspiration retrieval, hypothesis composition, and hypothesis ranking.The benchmark spans 12 disciplines and uses an automated framework for scalable, contamination-resistant construction.
  • LLMs show strong inspiration retrieval but only moderate hypothesis composition and ranking.The reported pattern indicates that models capture useful associations yet lack deeper integrative reasoning.
  • Improving composition and ranking could support high-quality hypothesis generation and advance automated scientific discovery.

Limitations

ResearchBench is currently limited in disciplinary coverage, fine-grained hypothesis discovery, external-feedback-based hypothesis updating, and training applications of its automatically collected labels.

  • ResearchBench covers only 12 disciplines because of budget and resource constraints, with selection driven primarily by data and resource availability.No discipline was excluded post hoc because it appeared incompatible with the theoretical decomposition.
  • The benchmark is not designed for fine-grained hypothesis discovery.Extending it would require extracting fine-grained hypotheses from literature and developing tailored composition metrics.
  • ResearchBench does not evaluate whether discovery systems can use external experimental feedback to update hypotheses.
  • Using automatically collected labels for model training remains underexplored.Further research is needed on training paradigms for scientific discovery.

A.1 Data Contamination Analysis

The benchmark addresses contamination by evaluating early model versions and testing a stricter post-July 2024 subset. Results remain close between the full and strict benchmarks, supporting the authors’ contamination-avoidance claim.

  • Contamination controls: Closed-source models are evaluated using their earliest available versions to reduce contamination risk.Model knowledge-cutoff and release dates are documented in Table 7.
  • Contamination controls: The stricter subset contains only papers published after July 2024, placing them farther from baseline model cutoff dates.All three benchmark tasks are rerun on this subset.
  • Benchmark subsets: The Original benchmark covers all 2024 papers, whereas Strict covers the post-July 2024 subset.Models marked † have cutoff dates strictly before July 2024 and theoretically cannot have seen the strict subset during pre-training.
  • Results: Results on the post-July 2024 subset are very close to results on the full 2024 benchmark.The authors characterize the differences as minor and consistent with normal sample variance.
  • Results: The authors interpret the stability between full and strict results as evidence that the benchmark method effectively avoids data contamination.The comparison spans inspiration retrieval, hypothesis composition, and hypothesis ranking.

A.2 Expert Evaluation Details and Guidelines

Expert validation assesses whether the automated decomposition correctly captures research questions, inspirations, and hypotheses, including whether inspirations are necessary and sufficient for recomposition. Five domain-specific PhD annotators perform the review, with assignments designed to limit cross-disciplinary evaluation.

  • Expert evaluation: Five PhD students from Physics, Chemistry, Materials Science, and Astronomy evaluate extracted research questions, inspirations, and hypotheses.They also assess whether inspirations are necessary and sufficient for recomposing the hypothesis.
  • Expert evaluation: Experts follow a three-step validation procedure after selecting benchmark papers within their own research expertise.The supplied procedure description begins with domain-specific paper selection.
  • Expert evaluation: Paper assignments are restricted to experts’ research domains because cross-disciplinary experts may not be qualified to evaluate unfamiliar papers.This assignment strategy means experts rarely review the same papers, limiting direct inter-annotator overlap.
  • Guidelines: The guidelines ask reviewers to assess whether the background question is correctly decomposed and whether the ground-truth hypothesis reflects the paper’s main proposal.Each assessment requires detailed analysis.
  • Guidelines: Reviewers evaluate whether each inspiration provides necessary conditions for reaching the ground-truth hypothesis.The guideline separately asks about Inspiration 2 and Inspiration 3.
  • Guidelines: Reviewers also judge whether the collected inspirations provide sufficient conditions for reaching the coarse-grained hypothesis.This criterion complements the inspiration-level necessity checks.

A.3 Prompt for Retrieving Inspirations

The prompt defines inspiration retrieval as identifying publications whose ideas can combine with a research question to form a novel, valid, and significant hypothesis. It emphasizes that useful inspirations may be distinct rather than semantically similar to the research question.

  • Inspiration retrieval selects a publication that can combine with the background research question to form a novel, valid, and significant hypothesis.The input includes a research question, background, existing methods, and several top-tier publication titles and abstracts.
  • Useful inspirations need not resemble the research question and may instead be distinct from it.The prompt argues that distinct knowledge can add information when combined with the background question.
  • Semantically similar inspirations may contribute little because they can duplicate the research question rather than add information.

A.4 Prompt for Evaluating Generated Hypothesis

The appendix specifies prompts for evaluating generated hypotheses, ranking competing candidates, and mutating, refining, or recombining hypotheses. It also illustrates the framework with a materials-science example and identifies ground-truth inspirations and a ranking error.

  • A.4 Prompt for Evaluating Generated Hypothesis: The generated-hypothesis evaluation prompt compares a proposed hypothesis with a ground-truth hypothesis by checking coverage of the ground-truth’s key points.The prompt uses a 6-point Likert-style Matched score from 5 to 0.
  • A.4 Prompt for Evaluating Generated Hypothesis: The scoring guidance allows multiple scores when a ground-truth hypothesis has few key points and the candidate nearly identically covers them.Extra key points and their quality determine the choice between the higher scores.
  • A.5 Prompt for Pairwise Ranking: The pairwise-ranking prompt asks which of two LLM-generated hypotheses is more valid and effective for a research question.The candidates are the ground-truth and negative hypotheses in the stated template.
  • A.5 Prompt for Pairwise Ranking: Ranking instructions ignore claimed expected performance because neither hypothesis has been experimentally tested.The prompt instead asks evaluators to focus on technical content and predicted effectiveness if tested.
  • A.5 Prompt for Pairwise Ranking: Ranking prioritizes the fundamental intrinsic idea rather than additional detail, complexity, comprehensiveness, or potential challenges.The prompt defines these extra attributes as neither advantages nor disadvantages.
  • A.6 Prompts for Mutate, Refine, and Recombine: The mutation, refinement, and recombination prompts decompose hypothesis proposal into stages involving a research question, prior methods, inspirations, and hypothesis generation.The prompts differ in whether the workflow is described as three or four steps.
  • A.7 Detailed Example of Framework: The detailed framework example shows the pipeline reading a target paper and producing an analysis plus an extracted hypothesis about a tunable AlCuFe quasicrystal absorber.The example concerns terahertz absorption and a grating structure on a gold substrate.
  • A.7 Detailed Example of Framework: The example links Dirac-semimetal photodetection and grating-based Fabry–Perot light trapping to the target absorber design, while identifying Inspirations 1 and 2 as ground truth.A separate ranking example reports GPT-4o incorrectly selected a negative hypothesis.
Loading 2503.21248v3…