PaperlayerView exact revision on arXiv

Source-linked AI summary

HypoKG: Evidence-Disciplined Biomedical Hypothesis Generation Beyond Endpoint Knowledge

Dominic Okonkwo, Adetayo Okunoye, Ismailcem Budak Arpinar

arXiv:2609.12260v1cs.CLcs.AIq-bio.QM

TL;DR

The paper asks whether LLMs use scientific evidence or generate plausible biomedical hypotheses from endpoint knowledge alone. It builds a controlled KEGG–Rhea–UniProt benchmark and finds that endpoint-only prompts score highest overall, while full paths improve evidence proportionality and shuffled paths weaken it.

  • Problem

    It is unresolved whether KG paths change LLM-generated hypotheses or models rely on parametric memory to produce endpoint-driven explanations, a concern because weakly supported mechanisms can mislead experimental design.

    Source: §1 · pp. 1–2

  • Method

    The authors build a KEGG–Rhea–UniProt graph benchmark of 550 source-to-disease paths and evaluate 13,200 hypotheses from six LLMs across four grounding conditions, including endpoint-only and full-path prompts.

    Source: §1 · p. 2, §3.5 · p. 4

  • Results

    Endpoint-only prompting achieved the highest aggregate scores, while full KG paths uniquely improved evidence proportionality; shuffling intermediates reduced C2 evidence proportionality by ∆=−0.793 (p < 0.001).

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that KGs can identify endpoint pairs absent from literature and provide mechanistic paths that guide how LLMs reason between them.

    Source: §6 · p. 9

    Evidence proportionality is a proxy for grounded reasoning and does not verify whether generated hypotheses are factually correct or experimentally validated.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) can generate biomedical hypotheses, but it remains unclear whether they truly reason from scientific evidence or simply produce convincing-sounding ideas. To study this, we combine three major biological databases: the Kyoto Encyclopedia of Genes and Genomes (KEGG), Rhea, and UniProt, into a unified biochemical knowledge graph and construct a benchmark of 550 paths connecting enzyme sources to rare disease endpoints, yielding 13,200 hypotheses from six LLMs under four conditions varying the biological information each model receives: source enzyme only, full biological path, or source and disease endpoint only. Hypotheses are scored using an expert-derived five-criterion rubric on a 1-5 scale per criterion. We find that models given both the source and disease endpoint often produce the highest-scoring hypotheses, showing that LLMs can generate compelling ideas from minimal information. However, these hypotheses are less grounded in the evidence. In contrast, models given the full biological path generate hypotheses more consistent with known mechanistic relationships. We call this evidence-disciplined reasoning. To confirm this effect, we shuffled intermediate path steps while keeping endpoints fixed. Evidence grounding dropped significantly (delta = -0.793, p < 0.001), confirming models genuinely used path structure during reasoning. Our findings show that knowledge graphs support hypothesis generation in two ways: they identify biological endpoint pairs absent from the literature, and their mechanistic paths guide how LLMs reason between them.

1 Introduction

The introduction asks whether LLMs use KG paths to reason mechanistically or generate plausible explanations from endpoints alone. It presents HypoKG as a controlled benchmark and reports a tension between aggregate quality and evidence grounding.

1 Introduction

  • Prior KG-grounded benchmarks do not isolate whether full mechanistic paths add value beyond revealing the source–disease pair.Endpoint-only prompting provides this comparison against path-conditioned generation.Source: §1 · p. 2
  • HypoKG combines KEGG, Rhea, and UniProt into a benchmark of 550 paths and 13,200 hypotheses from six LLMs under four grounding conditions.The conditions include source-only, two full-path variants, and endpoint-only prompting.Source: §1 · p. 2
  • Endpoint-only prompting achieved the highest total rubric scores, while full KG paths uniquely improved evidence proportionality.The authors call this evidence-disciplined reasoning and report that expert evaluation supports the effect.Source: §1 · p. 2
  • Shuffling intermediate nodes while holding endpoints fixed reduced C2 evidence proportionality by ∆=−0.793 (p < 0.001).This control tests whether gains depend on genuine intermediate path structure.Source: §1 · p. 2

2 Related Work

Prior work uses LLMs for hypothesis generation and biomedical KG reasoning, but HypoKG isolates how different forms of KG evidence affect open-ended hypotheses.

2 Related Work

  • Hypothesis-generation studies span open-domain, data-driven, and biomedical settings, while some human studies find greater judged novelty but feasibility trade-offs for LLM ideas.Other approaches combine causal KGs or literature with empirical data.Source: §2 · p. 2
  • Biomedical KGs support translational reasoning and precision medicine, while KG-grounded LLMs have been used for question answering, explainable generation, and relation prediction.Most prior work treats KG access as retrieval support rather than testing its effect on open-ended hypotheses.Source: §2 · p. 2
  • HypoKG’s endpoint-only ablation tests whether full mechanistic paths add value beyond revealing the same source–disease endpoints.Existing biomedical hypothesis benchmarks evaluate truthfulness, agent self-evaluation, or materials discovery rather than isolating this contrast.Source: §2 · p. 3

3 HypoKG Benchmark

HypoKG constructs a human biochemical KG benchmark and varies the evidence given to six LLMs, separating endpoint discovery from mechanistic path grounding.

3.1 Benchmark Overview

  • The benchmark uses 550 multi-hop paths connecting enzyme-kinetics sources to rare-disease endpoints, with six LLMs generating 13,200 hypotheses under four prompting conditions.The endpoint-only condition tests whether source–disease pairs support plausible hypotheses without intermediate path structure.Source: §3.1 · p. 3

3.2 Integrated Biochemical Knowledge Graph

  • The integrated human biochemical graph combines KEGG pathway topology, Rhea reaction detail, and UniProt protein annotations, and contains 17,449 nodes and 31,709 edges.The graph is restricted to human biochemistry to avoid contamination from model-organism pathways sharing enzyme nomenclature with human homologs.Source: §3.2 · p. 3

3.3 Cross-Domain Path Construction

  • A stratified selection produced 550 paths from constrained 3–12-node candidates, spanning 417 source entities and 366 diseases; experts judged none invalid.About 40% were directly supported; the remainder included interpretive ambiguity from high-degree annotation hubs, retained to test reasoning under incomplete evidence.Source: §3.3 · pp. 3–4

3.4 Literature Prevalence Annotation

  • PubMed co-citation analysis classified 520 of 550 source–disease connections as Novel (94.5%), placing the benchmark primarily in a low-evidence regime.The annotation used exact entity names followed by validated aliases.Source: §3.4 · p. 4

3.5 Generation Conditions

  • C1–C4 vary the grounding information, with C4 providing only source and disease endpoints and testing path value beyond endpoint discovery.All prompts require specific entities, a falsifiable experimental prediction, and grounding in established biochemistry.Source: §3.5 · p. 4

3.6 Model Suite and Generation Protocol

  • The six-model suite spans general-purpose, biomedical-specialized, proprietary, and open-weight systems, with one hypothesis per path and condition and a 512-token output limit.Truncated responses were retained and scored as generated; BioMistral-7B had the highest truncation rate at 3.9%.Source: §3.6 · pp. 4–5

4 Evaluation Framework

The evaluation combines entity-use diagnostics with an expert-derived rubric focused on whether mechanistic claims are proportionate to available evidence. Cross-judge scoring and blinded expert review assess the rubric, while a shuffled-path control tests whether models use intermediate path structure.

4.1 Automated KG Consistency Metrics

  • Entity-use metrics track source and disease mentions plus intermediate-entity coverage, but are treated as grounding diagnostics rather than measures of scientific quality.A hypothesis can mention many path entities yet remain mechanistically weak, or mention few while making a coherent focused claim.Source: §4.1 · p. 5

4.2 Expert-Derived Rubric

  • The five-criterion rubric scores relevance, mechanistic specificity, testability, novelty, and evidence proportionality from 1–5, yielding total scores from 5 to 25.Eight PhD-level experts informed the rubric; its criteria were set before the endpoint-only condition was introduced.Source: §4.2 · p. 5

4.3 Cross-Judge LLM Evaluation

  • Two LLM judges score each hypothesis, with no model judging its own output, and achieve Pearson r = 0.776 agreement on total scores.Agreement is lowest for evidence proportionality, reflecting the greater subjectivity of claim-to-evidence calibration.Source: §4.3 · p. 5

4.4 Human Expert Validation

  • Three PhD-level experts independently score evidence proportionality for all four condition outputs on a blinded, randomized sample of 55 paths.Within-path comparisons hold source and terminal disease constant, controlling for path difficulty and disease familiarity.Source: §4.4 · p. 5

4.5 Shuffled-Path Control

  • The shuffled-path control permutes intermediate nodes while keeping endpoints and full-path prompts fixed, testing whether ordered path structure affects evidence proportionality.Paths with fewer than three intermediate nodes are excluded, leaving 518 shuffled paths.Source: §4.5 · p. 6

5 Results

Endpoint-only prompting achieved the highest aggregate rubric score, while full-path conditions improved evidence proportionality; model capability and path complexity shaped the benefits of grounding. Human evaluation and shuffled-path controls support the finding that intermediate path structure disciplines mechanistic claims.

5.1 Endpoint Knowledge vs. KG Grounding

  • C4 scored highest overall (16.68 versus C2 at 15.14 and C3 at 14.96), while C2 and C3 significantly exceeded C4 on evidence proportionality (C2: +0.23; C3: +0.20).The endpoint pairs in C4 were discovered through KG construction and path sampling; the authors describe endpoint-only hypotheses as less constrained by mechanistic evidence.Source: §5.1 · p. 6, Table 3 · p. 7

5.2 KG Path Utilization Is Model-Dependent

  • C3–C1 gains ranged from 3.1 to 4.2 points for four stronger models, versus 0.05 for BioMistral-7B and 0.62 for Llama-3.3-70B.MedGemma-27B was the only model for which C3 outperformed C4; the authors suggest productive path use requires sufficient capacity and sometimes domain-specialized pretraining.Source: §5.2 · p. 7

5.3 Path Complexity Effects

  • C3–C1 gain peaked at crossing count 4 (+4.72), fell to +0.33 at count 8, and correlated negatively with crossing count (r = −0.223, p < 0.001).Qualitative inspection found local subgraph collapse: models focused on a strong local transition while ignoring the source or terminal disease.Source: §5.3 · pp. 7–8, Table 4 · p. 8

5.4 Human Expert Validation

  • Experts ranked conditions C2 > C3 > C4 > C1, with path-grounded conditions beating C4 on 90.3% of paths and unanimous C4 speculation flags on 50/55 paths.Three experts assessed a stratified sample of 55 paths in a blinded within-path design.Source: Table 5 · p. 8, §5.4 · p. 8

5.5 Shuffled-Path Control

  • Shuffling intermediate nodes reduced evidence proportionality (C2: −0.793; C3: −0.592; both p < 0.001) with endpoints fixed, while novelty did not significantly change.The grounding drop varied by model; BioMistral-7B showed a near-zero, non-significant C3 drop (−0.023, p = 0.242).Source: §5.5 · pp. 8–9, Table 6 · p. 8

6 Conclusion

HypoKG evaluates biochemical hypothesis generation across controlled grounding conditions and finds that endpoint-only prompts score highest overall, while full paths improve evidence proportionality. Human validation and shuffled-path controls support the conclusion that intermediate path structure shapes mechanistic claims.

6 Conclusion

  • Full KG paths uniquely improve evidence proportionality, while endpoint-only prompting achieves the highest aggregate scores.Human validation and shuffled-path controls support that the effect reflects use of intermediate path structure.Source: §6 · p. 9
  • HypoKG benchmarks generation across 550 multi-hop paths, six LLMs, and four controlled grounding conditions.The benchmark focuses on biochemical hypotheses connecting sources to rare-disease endpoints.Source: §6 · p. 9
  • HypoKG frames knowledge graphs as both a means of endpoint discovery and an epistemic constraint on biomedical hypothesis generation.Source: §6 · p. 9

7 Limitations

The evaluation is bounded by its expert-study scale, graph and path-selection scope, and the limits of rubric-based grounding measures. The appendix prompt operationalizes scoring across five criteria while distinguishing what evidence each generator received.

7 Limitations

  • Expert evaluation covered 55 paths, and inter-judge agreement was lowest for evidence proportionality (r = 0.668).The three PhD-level domain collaborators participated voluntarily; no personal data was collected and IRB review was not required under institutional guidelines.Source: §7 · p. 9
  • The benchmark is restricted to human biochemistry and uses constrained shortest-path traversal, which may surface different evidence than alternative path strategies.The integrated graph omits resources such as SIDER and HPO.Source: §7 · p. 9
  • Results may not generalize to models released after the data-collection cutoff, and endpoint pairs may overlap with pretraining data.The 94.5% zero-citation rate and shuffled-path controls reduce, but do not eliminate, this concern.Source: §7 · p. 9
  • Aggregate rubric scores may miss dimensions of hypothesis quality relevant to practicing scientists, and evidence proportionality does not verify factual correctness or experimental validation.Source: §7 · p. 9

A.2 Judge System Prompt

  • The judge prompt scores five criteria, including evidence proportionality, with that criterion conditioned on whether the generator saw the full path.It distinguishes path-informed scoring from cases where no KG path was shown.Source: §A.2 · p. 13

Condition 4 (C4): Endpoint-Only Ablation

C4 tests whether source and disease names alone can support hypothesis generation without showing intermediate path structure. Its prompts require a specific, testable mechanistic connection, and the benchmark's evaluation rubric was developed through expert calibration.

Condition 4 (C4): Endpoint-Only Ablation

  • C4 gives the model only the source enzyme and target disease, requiring a specific mechanistic connection and a falsifiable prediction grounded in known biochemistry.The prompt also requires naming the source enzyme, connecting specifically to the target disease, and limiting the response to 150 words.Source: Condition 4 (C4): Endpoint-Only Ablation · p. 12

B Expert-Derived Evaluation Rubric

  • The five-criterion rubric was derived from structured calibration sessions with eight PhD-level experts in biochemistry and metabolic disease.Experts independently scored hypotheses for a held-out path under all four conditions and provided written justifications.Source: B Expert-Derived Evaluation Rubric · p. 13
  • Total rubric scores range from 5–25 points, with scores of 20–25 labeled strong and scores below 10 labeled poor.Scores of 15–19 are usable but need refinement; 10–14 are weak or partial.Source: B Expert-Derived Evaluation Rubric · p. 13

Criterion 1: Path/Task Relevance

The rubric separates addressing the assigned source-to-disease problem from proposing a specific, biochemically coherent mechanism. Expert feedback motivated a dedicated relevance criterion and emphasized concrete reaction-level anchors.

Criterion 1: Path/Task Relevance

  • Path/task relevance measures whether a hypothesis connects the assigned source to the terminal disease rather than drifting to another question.All six experts who saw the C1 disease-drift example flagged it as well-formed but path-disconnected, motivating this dedicated criterion, which was new to KG evaluation.Source: Criterion 1: Path/Task Relevance · pp. 13–14

Criterion 2: Mechanistic Specificity and Consistency

  • Mechanistic specificity and consistency rewards named biochemical entities and a coherent mechanism consistent with known biochemistry.Experts highlighted AGXT → glyoxylate → oxalate as a mechanistic anchor and penalized pathway-level language.Source: Criterion 2: Mechanistic Specificity… · p. 14

Criterion 3: Experimental Testability and Operationalization

The rubric operationalizes experimental testability and non-trivial novelty as distinct dimensions of hypothesis quality. Expert feedback shaped both criteria, particularly the need to distinguish actionable predictions from unsupported precision and textbook restatements.

Criterion 3: Experimental Testability and Operationalization

  • Testability requires a concrete perturbation, model system, measurable outcome, and falsification condition; its five-point scale rewards complete designs and penalizes vague or untestable predictions.Experts preferred AGXT knockdown, glyoxylate challenge, and dose-response testing, and criticized unjustified numerical precision.Source: Criterion 3: Experimental Testability… · p. 14
  • Testability scores distinguish complete experimental proposals from designs missing elements, predictions lacking specifics, and unjustified precise numbers stated as fact.The scale ranges from 1 for no testable prediction to 5 for a specified experiment with justified or appropriately hedged quantitative claims.Source: Criterion 3: Experimental Testability… · p. 14

Criterion 4: Non-Trivial Novelty and Knowledge Advancement

  • The novelty criterion asks whether a hypothesis adds a new relationship, mechanism, modifier, or experimental angle beyond established knowledge.Five of six experts independently called a calibration hypothesis correct but trivial or textbook, motivating a low score for restated biology.Source: Criterion 4: Non-Trivial Novelty and… · pp. 14–15

Criterion 5: Evidence Proportionality and Full-Path Use

Evidence proportionality evaluates whether hypotheses use path evidence intelligently and calibrate claim strength accordingly. Five examples contrast path-irrelevant or overconfident proposals with increasingly grounded, mechanistically specific hypotheses.

Criterion 5: Evidence Proportionality and Full-Path Use

Example 1: Path-irrelevant but fluent

  • The source-only example links glutathione reductase to age-related macular degeneration rather than the target disease, earning a total score of 11 and illustrating disease drift.Its criterion scores were Rel=1, Mech=2, Test=4, Nov=3, and Prop=1.Source: Example 1: Path-irrelevant but fluent · p. 15

Example 2: Correct but textbook

  • The AGXT example correctly explains Primary Hyperoxaluria Type 1 but restates textbook biology, uses the strongest path edge, and discards CDO1.It receives a total score of 14; the proposal includes specific activity and urinary oxalate predictions.Source: Example 2: Correct but textbook · p. 15

Example 3: Strong but imperfect

  • The strong-but-imperfect example proposes a dose-response test of AGXT knockdown and earns 18, but leaves cell type unspecified and abandons upstream CDO1.Its rationale treats the cyanoamino acid connection as weaker than the AGXT–disease relationship.Source: Example 3: Strong but imperfect · pp. 15–16

Example 4: Endpoint-only, plausible but overconfident

  • The endpoint-only CDO1 proposal plausibly connects deficiency to Primary Hyperoxaluria Type 1, but its mechanism is overconfident without path evidence and scores 15.The example notes that the bridge should instead be framed as speculative.Source: Example 4: Endpoint-only, plausible… · p. 16

Example 5: Excellent

  • The highest-scoring example proposes CDO1 as a modifier of AGXT-deficient disease, uses both path elements, and remains appropriately hedged, earning 25.It proposes a concrete knockdown experiment, LC-MS oxalate measurement, and pyridoxine rescue; the example is not experimental validation.Source: Example 5: Excellent · p. 16

C Additional Results

Additional analyses test whether models use intermediate path structure, examine judge agreement across rubric criteria, and compare entity-mention grounding across prompting conditions.

C Additional Results

  • 0.68–1.11 drops under shuffling were significant for stronger models in C2 and C3, while BioMistral-7B’s C3 drop was −0.023 and non-significant (p = 0.242).Llama-3.3-70B showed significant drops despite weak overall grounding gains; the test permuted intermediate nodes while holding endpoints fixed.Source: C Additional Results · p. 16, Table 7 · p. 17
  • Agreement was highest for experimental testability (r = 0.839) and path/task relevance (r = 0.813), and lowest for evidence proportionality (r = 0.668).The table reports secondary-judge agreement with the primary judge across 13,200 hypotheses.Source: C Additional Results · p. 16, Table 8 · p. 17
  • C1 showed near-zero terminal grounding, C4 high terminal grounding, and C3 the highest path-entity coverage; these mention metrics are diagnostics, not scientific-quality measures.The metrics distinguish source and disease mentions from intermediate-entity coverage and compare the information used across conditions.Source: C Additional Results · p. 17, Table 9 · p. 17

Related papers

Loading 2609.12260v1…