PaperlayerView exact revision on arXiv

Source-linked AI summary

ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure

A. J. Lew, Y. Cao, M. J. Buehler

arXiv:2605.30284v1cs.AI

TL;DR

Scientific-discovery benchmarks have largely tested retrieval or established reasoning rather than innovative hypothesis generation from minimally specified research questions. ProjectionBench progressively reveals context and scores atomic-claim alignment with ground-truth conclusions, finding stronger performance from newer models and GPT-5.4 retaining about 0.70 F1 under minimal context.

  • Problem

    Existing LLM benchmarks leave innovative scientific reasoning insufficiently evaluated, despite discovery requiring both novelty and grounding in established knowledge.

    Source: §1 · pp. 1–2

  • Method

    The benchmark progressively reveals research context from topic and question to null hypothesis and experimental procedure, then grades projected atomic claims against ground truth using GPT-5-assisted alignment, precision, recall, and F1.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    GPT-5.4 and Gemini 3.1 Pro Preview consistently outperform earlier counterparts, while GPT-5.4 maintains F1 ≈0.70 under minimal context.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The framework evaluates both innovative low-context projections and grounded high-context reasoning across progressively disclosed scientific information.

    Source: §6 · p. 9

    Using GPT-5 as judge for GPT-family models may introduce evaluation bias.

    Source: §6 · p. 9

Abstract

from arXiv · show

Scientific discovery is an inherently creative and uncertain process, requiring reasoning beyond the recall of known knowledge. While many benchmarks have been proposed to evaluate large language model (LLM) performance on deep research tasks via multi-hop retrieval, their innovative reasoning abilities essential for true scientific discovery remain largely untested. We introduce a benchmark framework for evaluating model performance in scientific discovery and reasoning, building up from a raw problem to the classical null hypothesis test. In our framework, models initially receive only the topic and research question from a recent paper, with technical details progressively revealed. At each stage of information disclosure, the model is tasked with generating hypotheses that address the research question, which is compared with the conclusions from the original paper and evaluated via automated semantic similarity of constituent atomic claims. This progressive evaluation of semantic divergence from ground-truth conclusions enables assessment of a model's innovativeness (under minimal information) to grounded reasoning capabilities (under full experimental details), both critical for using LLMs for scientific discovery purposes. Our framework provides a foundation for systematically evaluating scientific reasoning and discovery capabilities in LLMs, crucial for advancing the development of next-generation AI scientist/co-scientist systems. Specifically, here we evaluate GPT-5, GPT-5.4, Gemini 2.5 pro, and Gemini 3.1 pro preview across 45 papers spanning bioactive materials, mechanical materials, and nanomaterials. We find that GPT-5.4 and Gemini 3.1 pro outperform their previous generation counterparts as expected, and GPT-5.4 in particular maintains 0.7 F1 score alignment with ground truth conclusions even under minimal context.

1 Introduction

Scientific-discovery evaluation remains limited because existing benchmarks emphasize retrieval, citation alignment, curated tasks, or intensive analyses rather than lightweight discovery from early-stage research questions. The paper proposes a scalable benchmark that progressively reveals context from topic and question to experimental details, evaluating both innovative hypotheses and grounded conclusions.

  • Scientific discovery requires novelty grounded in established knowledge, because creativity without correctness produces unsupported claims rather than discovery.Source: §1 · p. 1
  • Existing benchmarks mainly test factual recall, retrieval, citation quality, textbook reasoning, or curated data-driven hypotheses, leaving early-stage discovery insufficiently measured.Source: §1 · pp. 1–2
  • The benchmark progressively discloses information from only a topic and research question to the null hypothesis and experimental procedures, testing innovation and reasoning together.Source: §1 · p. 2
  • The framework targets live, scalable evaluation rather than relying only on established knowledge, expert annotations, retrieved literature, or heavily curated scenarios.Source: §1 · p. 2

2 Discovery Task

The discovery task emulates null-hypothesis testing by asking models to project research findings while progressively increasing the available context. This design evaluates creative hypothesis generation with minimal information and accurate inference from experimental procedures.

  • Models receive a variable amount of research context and project the solution or findings of a research question within a null-hypothesis-testing framework.Source: §2 · p. 3
  • Context ranges from only the topic and research question, through a base null hypothesis, to a specific experimental procedure.Source: §2 · p. 3
  • The benchmark assesses whether models generate novel insights with minimal context and accurately infer conclusions when procedures are provided.Source: §2 · p. 3

3 Automated Grading Method

The grading method decomposes ground-truth and projected results into atomic claims, aligns them with an LLM judge, and aggregates precision and recall into F1 and disclosure-level AUC. Toy-claim tests verify that scores track the fraction of manuscript information provided.

3.1 Granular Grading Approach

  • Ground-truth and projected results are decomposed into atomic relationships between manipulated independent variables and measured dependent variables.Source: §3.1 · p. 3, Figure 1 · p. 4
  • Projected claims are matched to ground-truth claims, while unmatched projected claims are separately extracted as extraneous claims.Source: §3.1 · p. 4

3.2 Calculating Grade Score

  • GPT-5 judges claim alignment, with evaluations repeated after reversing claim order to reduce positional bias.Source: §3.2 · p. 4
  • Precision and recall are combined into F1, while AUC aggregates F1 across disclosure levels as the final evaluation metric.Source: §3.2 · pp. 4–5
  • Extraneous projections are compared against all ground-truth claims using their average alignment, while positive and negative alignment scores define true and false positives.Source: §3.2 · pp. 4–5

3.3 Validating Grading Approach

  • Toy-claim validation shows that scores approach one with nearly complete manuscripts, approach zero with no manuscript, and correlate with provided information.Source: §3.3 · p. 5

4 Dataset Curation

The benchmark uses a live, contamination-conscious corpus of 45 recent open-access materials-science papers and extracts linked experimental information in staged waves. Extraction proceeds from factual details to hypotheses, null hypotheses, research questions, and results.

  • The dataset contains 45 open-access Springer Nature manuscripts from bioactive materials, nanomaterials, and mechanical materials, with 15 papers per category.Source: §4 · p. 5
  • Papers are selected from the previous six months to reduce training contamination for the evaluated models.Source: §4 · p. 5
  • Full texts are parsed in three waves covering title, topic, procedure, hypothesis, null hypothesis, research question, and results.Source: §4 · p. 5
  • Extraction begins with factual details, then uses those details to constrain hypothesis extraction and avoid incorporating ancillary or future-application content.Source: §4 · p. 6
  • The extracted hypotheses and procedures guide extraction of connected null hypotheses, research questions, and results so conclusions address the question through experimental testing.Source: §4 · p. 6

5 Experiment and Results

The benchmark compares four models across context levels and materials domains, showing stronger recent models, context-dependent gains, and substantial manuscript-level difficulty variation.

  • GPT-5.4 and Gemini 3.1 Pro Preview consistently outperform their earlier counterparts across all provided-context levels.Source: §5 · pp. 6–7
  • Adding a null hypothesis improves performance more than subsequently adding the experimental procedure, indicating diminishing returns from further context.Source: §5 · p. 7
  • F1 ≈0.70: GPT-5.4 maintains strong minimal-context performance, while Gemini 2.5 Pro needs fuller experimental details for comparable results.Source: §5 · p. 7
  • Bioactive manuscripts score higher than mechanical and nanomaterial manuscripts, with smaller relative gains from additional context.Source: §5 · p. 7
  • Mechanical manuscripts show a context-dependent model trade-off: Gemini 3.1 Pro Preview surpasses GPT-5 at high context but performs slightly lower at low context.Source: §5 · p. 8
  • Large standard deviations reflect heterogeneous manuscript difficulty; bioactive scores cluster high, mechanical scores vary widely, and nanomaterials retain a higher performance floor.Source: §5 · p. 8, Figure 4 · p. 8
  • A mechanical-manuscript example shows GPT-5.4 moving from partially aligned minimal-context projections to matching all ground-truth relationships with full experimental details.Source: §5 · p. 8, Table 3 · p. 9

6 Conclusions

The paper presents ProjectionBench as an initial framework for evaluating whether models can project scientific outcomes rather than merely retrieve established knowledge. It concludes that context-sensitive evaluation can guide assessment of both innovative low-context projection and high-context reasoning, while acknowledging judge bias and future expansion needs.

  • The framework addresses the greater challenge of anticipating novel research outcomes compared with recall-based, exam-style benchmarks.Source: §6 · p. 9
  • ProjectionBench evaluates plausible scientific outcome projection, distinguishing low-context innovative generation from high-context synthesis and reasoning.Source: §6 · p. 9
  • Using GPT-5 as judge for GPT-family models may introduce bias, motivating future cross-family evaluation with independent models.Source: §6 · p. 9
  • Future work will broaden scientific domains and model coverage while refining context granularity and reasoning-process evaluation.Source: §6 · p. 9

Prompt 1: Scientific Projection Task

The prompt progressively constructs the scientific projection task by adding hypothesis and experimental-procedure context to the topic and research question. Models are instructed to state projected results as qualitative relationships between independent and dependent variables.

Prompt 2: Ground Truth Claim Extraction

Prompt 2 extracts mutually exclusive relationship claims from a scientific passage, representing each claim with an independent variable, relationship, and dependent variable.

Prompt 3: Analogous Claim Extraction

This prompt extracts mutually exclusive relationship claims analogous to provided examples, using experimental variables and their qualitative relationships as the target structure.

Prompt 4: Remaining Claim Extraction

This section describes the extraction of structured scientific and bibliographic content for evaluating alignment between predicted and ground-truth statements. It covers staged document parsing and associated scoring-table metadata, without reporting score values or model comparisons.

Prompt 4: Remaining Claim Extraction

  • Prompt 4: Remaining Claim Extraction distinguishes manipulated independent variables from measured dependent variables and asks whether drafts omit relationship claims.The prompt requests up to three mutually exclusive missing claims in JSON, or an empty claims list when none are missing.Source: Prompt 4: Remaining Claim Extraction · p. 13

Prompt 5: Alignment Rubric

  • Prompt 5: Alignment Rubric defines ratings from -1 to 1 for misaligned, uncorrelated, and aligned scientific statements.Misalignment indicates opposing effect directions, uncorrelated statements describe different effects, and alignment indicates supporting relationships.Source: Prompt 5: Alignment Rubric · p. 13

Prompt 6: Document Parsing

  • The document-parsing workflow applies wave-based extraction, with hypothesis extraction following experimental-procedure extraction and null-hypothesis extraction following the alternative hypothesis.The staged implementation labels these steps as Wave 1, Wave 2, and Wave 3.Source: Prompt 6: Document Parsing · pp. 13–14
  • Prompt 6: Document Parsing first extracts a study title and summarizes its main topic in fewer than five words.These outputs are generated from each scientific passage using staged prompts.Source: Prompt 6: Document Parsing · pp. 13–14
  • Prompt 6: Document Parsing converts each passage into a numbered objective procedure that excludes results and subjective qualifications.The experimental extraction is framed as factual physical instructions.Source: Prompt 6: Document Parsing · p. 14
  • Prompt 6: Document Parsing extracts a falsifiable hypothesis as an If/When CONDITION then OBSERVABLE statement and formulates its unbiased null counterpart.Both prompts focus on qualitative relationships between variables and require concise objective statements.Source: Prompt 6: Document Parsing · p. 14
  • Prompt 6: Document Parsing separately extracts the broad research question in fewer than 10 words and the new key result as a qualitative relationship statement.The result prompt asks for the existence and qualitative extent of each relationship in the hypothesis.Source: Prompt 6: Document Parsing · pp. 14–15
  • The document lists a study on EPS-secreting microbes and their applications in microbial fuel cells and beyond.Source: Prompt 6: Document Parsing · p. 18
  • It also lists research on ZnO/NiO and ZnO/CuO heterostructures for high-performance solar-cell applications.Source: Prompt 6: Document Parsing · p. 18
  • Another extracted entry concerns biomass production and antioxidant dynamics in Ceratophyllum demersum L. using multi-walled carbon nanotubes and machine-learning models.Source: Prompt 6: Document Parsing · p. 18
  • A scoring-results table is identified in the parsed document.Source: Table 5 · p. 18
  • The supplied parsing passages identify table structure but do not report numerical scores or comparative outcomes.Source: Table 5 · p. 18, Prompt 6: Document Parsing · p. 19
  • The table metadata includes fields for model, category, context, and amount.Source: Prompt 6: Document Parsing · p. 19

Related papers

Loading 2605.30284v1…