PaperlayerView exact revision on arXiv

Source-linked AI summary

HyGRAIL: Cost-Aware and Evidence-Grounded Scientific Hypothesis Discovery over Knowledge Graphs

Yihang Sun, Zhihan Zhu, Zhiyuan Jiang, Jingyi Ge, Zixuan Li, Jiaxuan You

arXiv:2609.02056v1cs.CL

TL;DR

Incomplete scientific knowledge graphs leave plausible typed links undiscovered, but sparse candidates make graph-only decisions unreliable for ambiguous cases and exhaustive LLM review too costly and weakly grounded. HyGRAIL combines GNN triage with retrieved, naturalized graph evidence and LLM review for hard hypotheses. On MatKG, it achieves the best reported verification performance while reducing unnecessary LLM calls.

  • Problem

    Sparse discovery-relevant links, unreliable graph-only decisions for ambiguous cases, and costly, weakly grounded exhaustive LLM review complicate hypothesis verification over incomplete scientific knowledge graphs.

  • Method

    HyGRAIL uses a heterogeneous GNN to triage candidates, routes a validation-calibrated ambiguous region to LLM review, and supplies retrieved node-level and multi-hop graph evidence in naturalized form.

  • Results

    0.429 F1 is HyGRAIL’s best MatKG result, improving over TSH by 0.242 F1 points and over the corresponding R-GCN-only baseline by 0.322.

  • Takeaways & Limitations

    Retrieved graph evidence is crucial for reliable verification, while compact two-sided evidence and GNN triage support evidence-grounded prioritization with fewer unnecessary LLM calls.

  • Takeaways & Limitations

    Under the closed-world protocol, accepted unobserved links remain candidates for expert or experimental validation rather than definitive scientific facts.

Abstract

from arXiv · show

Scientific knowledge graphs organize entities and relations extracted from scientific literature, but they remain inherently incomplete. Missing typed links in such graphs can therefore represent plausible scientific hypotheses, such as unexplored associations between materials and applications. However, scientific hypothesis discovery is challenging because true discoveries are extremely sparse among typed candidate pairs: graph neural networks (GNNs) are efficient but unreliable for ambiguous cases, while large language models (LLMs) are knowledgeable but too costly to apply exhaustively and are not naturally grounded in graph structures. We propose HyGRAIL, a cost-aware and evidence-grounded framework that combines heterogeneous GNN triage with LLM-based hypothesis review. HyGRAIL first uses a GNN to score candidate hypotheses and identify a validation-calibrated ambiguous region, routing only graph-uncertain cases to LLM review. For each routed hypothesis, HyGRAIL retrieves node-level associations and multi-hop relational paths from the knowledge graph (KG), then converts this structured evidence into natural language through template-based or LLM-based naturalization. An LLM review agent finally judges each hard hypothesis using the naturalized evidence and validation-selected decision criteria. On MatKG, HyGRAIL achieves the best F1 score of 0.429, improving over the strongest prior baseline by 0.242 F1 points and over the GNN-only baseline by 0.322. Meanwhile, GNN triage reduces the LLM call rate by 54.36% on average. Ablation studies further show that retrieved graph evidence is crucial for reliable hypothesis verification and that compact, two-sided evidence is more effective than simply increasing retrieval quantity.

1 Introduction

Scientific hypothesis discovery over incomplete knowledge graphs is difficult because valid typed links are sparse, graph-only models are unreliable for ambiguous cases, and exhaustive LLM review is costly and weakly grounded in graph structure. HyGRAIL addresses these challenges by routing graph-uncertain hypotheses to evidence-grounded LLM review, achieving stronger verification while reducing LLM usage.

  • Motivation: Typed candidate pairs vastly outnumber discovery-relevant links, making scientific hypothesis discovery statistically difficult and practically high-stakes.False positives can waste expert attention or experimental resources, while false negatives may overlook promising discoveries.
  • Motivation: GNNs efficiently exploit heterogeneous graph structure but remain unreliable as final arbiters for hypotheses in ambiguous score regions.Graph-only predictions can mix positive and negative hypotheses in the middle of the score distribution.
  • Motivation: LLMs provide broad knowledge and language reasoning but are too costly for exhaustive candidate review and are not naturally designed for raw graph structures.Retrieval-based grounding is motivated by their reliance on incomplete parametric memory rather than explicit evidence.
  • HyGRAIL: HyGRAIL combines heterogeneous GNN triage with hypothesis-guided graph evidence retrieval, evidence naturalization, and LLM-based review.The framework routes only hypotheses in a validation-calibrated ambiguous region to the LLM while handling easier cases with the GNN.
  • Results: 0.429 F1 is HyGRAIL’s best reported score, while average LLM call rate falls by 54.36%.The result is reported for MatKG, with retrieved graph evidence and two-sided endpoint context identified as crucial in ablations.

2 Task Formulation

The task represents scientific knowledge graphs as weighted heterogeneous graphs and frames discovery as prediction over typed missing links. Evaluation uses a closed-world protocol, so unobserved links are benchmark negatives but not necessarily scientifically false.

  • 2.1 Scientific Knowledge Graphs: A scientific KG is modeled as a weighted heterogeneous graph whose nodes and edges have types and whose edges carry paper-derived support counts.The support count records how many scientific papers support each represented association.
  • 2.2 Hypotheses as Typed Missing Links: Discovery targets are specified by a subset of edge types, and each candidate hypothesis is a typed node pair whose endpoint types match its relation.A CHM–APL hypothesis states that a chemical may be useful for a particular application.
  • 2.2 Hypotheses as Typed Missing Links: For each discovery relation, the hypothesis space contains node pairs with the relation’s prescribed endpoint types; the full space combines these relation-specific spaces.The endpoint-type mapping determines which pairs are eligible for each relation.
  • 2.2 Hypotheses as Typed Missing Links: These hypothesis spaces are highly sparse, making exhaustive manual or LLM review impractical and graph-only prediction unreliable for ambiguous hypotheses.Only a small fraction of typed candidate pairs are observed as positive links.
  • 2.3 Closed-World Evaluation and Candidate Construction: Observed edges of target types are treated as positive hypotheses, while sampled unlinked typed pairs from the same hypothesis space are treated as negatives.The labels are defined as y_h = 1 for observed hypothesis edges and y_h = 0 for sampled unlinked candidates.
  • 2.3 Closed-World Evaluation and Candidate Construction: The closed-world protocol follows standard link-prediction practice but does not establish that an unobserved edge is scientifically false.Held-out observed edges serve as evaluation positives, whereas newly accepted unobserved links require expert or experimental validation.

3 Method

HyGRAIL combines heterogeneous GNN triage with graph-evidence retrieval, naturalization, and LLM review. Validation-calibrated thresholds reserve LLM inference for ambiguous hypotheses while direct GNN predictions handle confident cases.

  • GNN-based Hypothesis Triage: HyGRAIL scores typed candidate hypotheses with a heterogeneous GNN and identifies an ambiguous score region where graph-only predictions are unreliable.The framework uses graph representation learning to triage candidates before review.
  • GNN-based Hypothesis Triage: Validation selects the smallest interval whose lower and upper tails meet preset negative- and positive-purity requirements, minimizing routed validation hypotheses.The interval is chosen using validation scores and the KG-specific positive–negative ratio.
  • GNN-based Hypothesis Triage: At inference, HyGRAIL predicts low-score hypotheses negative, high-score hypotheses positive, and routes only interval members to the LLM review agent.This triage reserves costly LLM inference for cases the GNN cannot confidently resolve.
  • Hypothesis-Guided Graph Evidence Retrieval: For routed hypotheses, HyGRAIL retrieves node-level associations and short 2-hop or 3-hop paths as complementary structured evidence.Node evidence characterizes endpoint context, while paths capture relational connections between endpoints.
  • Evidence Naturalization: Retrieved graph evidence is naturalized into concise language using deterministic templates or an LLM before hypothesis review.Auto-Naturalization preserves support-count and normalized-weight categories, whereas LLM-Naturalization organizes related evidence and possible support or weakening.
  • LLM Hypothesis Review Agent: The LLM review agent judges routed hypotheses from naturalized evidence using a binary decision, confidence score, and validation-selected acceptance threshold.A hypothesis is accepted only when the decision is True and its confidence reaches γ.

4 Experimental Setup

HyGRAIL is evaluated on a sampled MatKG materials-science subgraph using temporally ordered splits and highly imbalanced negative sampling. Experiments compare heterogeneous GNN backbones, open-weight LLM reviewers, and evidence-removal or hybrid baselines.

  • Dataset: MatKG evaluation uses a 3,000-node materials-science subgraph, seven hypothesis types, and edge support counts derived from literature evidence.The sample approximately preserves graph density and node- and edge-type distributions.
  • Data Splits: Observed positive edges are split chronologically 7:1:2, while each split samples negatives at a 1:20 positive-to-negative ratio.Timestamps use the oldest publication contributing to each edge count.
  • Models and Baselines: The study compares HeteroConv, HGT, and R-GCN backbones, four open-weight LLM reviewers, and variants including HyGRAIL-no_evidence and TSH.The LLMs span Qwen3 and Ministral families at approximately 3B, 4B, and 14B scales.

5 Results and Analysis

HyGRAIL achieves strong MatKG hypothesis-verification performance while reducing LLM review through GNN triage. Ablations show that retrieved, compact, two-sided evidence improves verification reliability.

  • Main Results: 0.429 F1 is HyGRAIL’s best MatKG result, improving over TSH by 0.242, the best KG-FM result by 0.256, and the R-GCN-only baseline by 0.322.The result uses R-GCN and Qwen3-4B.
  • Main Results: Retrieved evidence improves F1 over the evidence-free variant across reviewers, with gains reaching 0.313 and 0.215 points for R-GCN with Qwen3-4B and Qwen3-14B.Under HeteroConv, gains range from 0.035–0.066 across four reviewers.
  • Main Results: HyGRAIL-Auto is generally stronger than HyGRAIL-LLM, indicating that template-based naturalization yields more concise and controlled evidence descriptions.The authors therefore support combining GNN triage with naturalized graph evidence for LLM review.
  • LLM Call Reduction: 45.64% is the average LLM call rate after GNN triage, which filters confident low- and high-score hypotheses before review.The paper characterizes triage as both predictive and cost-controlling.
  • Ablation Study: The default evidence budget improves F1 from 0.415 to 0.460 for Qwen3-4B and from 0.216 to 0.278 for Qwen3-14B, whereas larger budgets produce unstable gains.Qwen3-4B falls below default-budget performance at all larger budgets, while Qwen3-14B initially drops before benefiting at larger budgets.
  • Ablation Study: Full two-sided evidence improves F1 over one-sided evidence by 0.222 for Qwen3-14B and 0.152 for Qwen3-4B.The comparison keeps total node-level evidence unchanged and supports complementary endpoint context.

6 Related Work

Related work uses scientific knowledge graphs to structure fragmented literature and support hypothesis generation. HyGRAIL differs by retrieving and naturalizing typed graph evidence for cost-aware verification of specific hypotheses.

  • Scientific KGs and Hypothesis Discovery: Scientific knowledge graphs structure entities and relations from literature or curated databases across biomedical, scholarly, and materials-science domains.These resources support graph- and literature-based hypothesis generation.
  • Scientific KGs and Hypothesis Discovery: HyGRAIL targets cost-aware verification of specific scientific hypotheses rather than question answering, text generation, or free-form reasoning.Its approach retrieves and naturalizes typed graph evidence.

7 Conclusion

HyGRAIL combines GNN triage, graph-evidence retrieval, evidence naturalization, and LLM review to improve hypothesis verification while reducing unnecessary LLM calls. Experiments and ablations support retrieved evidence and evidence-grounded prioritization for expert inspection or experimental validation.

  • HyGRAIL combines GNN-based triage, hypothesis-guided graph evidence retrieval, evidence naturalization, and LLM-based review.
  • HyGRAIL improves hypothesis verification while reducing unnecessary LLM calls.
  • Retrieved graph evidence is crucial for reliable hypothesis verification.
  • HyGRAIL supports evidence-grounded hypothesis prioritization for expert inspection or experimental validation.

8 Limitations

The study identifies limitations involving its closed-world evaluation protocol, predefined schema-dependent retrieval configuration, and sensitivity of LLM review to model and calibration choices.

  • The closed-world protocol treats held-out observed edges as positives and sampled unlinked pairs as negatives.An unobserved edge does not necessarily imply scientific invalidity.
  • Accepted hypotheses should be treated as candidates for expert inspection or experimental validation, not definitive scientific facts.
  • Predefined hypothesis types and evidence edge-type sets may require lightweight domain-specific configuration when transferring HyGRAIL to a new KG schema.
  • LLM review may be affected by model choice, prompting details, and confidence calibration.Further calibration or ensemble-based review could improve robustness.

9 Ethical Considerations

The framework is intended to assist scientific hypothesis prioritization without replacing expert judgment or experimental validation. The study uses public scientific artifacts and open-weight models and identifies overreliance on model outputs as a standard concern.

  • HyGRAIL is intended to assist scientific hypothesis prioritization rather than replace expert judgment or experimental validation.
  • The study does not involve human subjects, personal data, private user information, or demographic attributes.
  • Accepted hypotheses are model-generated candidates that should be interpreted as suggestions for further expert review.
  • The study uses publicly available scientific artifacts and open-weight models only for research evaluation.
  • Automated scientific decision support carries a concern of overreliance on model outputs without domain expert validation.

A.1 Statistics of the Sampled MatKG Subgraph

The evaluation uses a sampled MatKG subgraph with preserved structural distributions, timestamp-based data splits, typed negative sampling, and predefined evidence edge-type sets.

  • A 3,000-node MatKG subgraph approximately preserves graph density, node type distribution, and edge type distribution.Its graph density is 1.097 × 10−3.
  • Table 4 reports node type statistics for the sampled MatKG subgraph.
  • Observed positive edges are split into training, validation, and test sets in a 7:1:2 ratio according to edge timestamps.
  • Negative examples are sampled from unlinked typed node pairs at a positive-to-negative ratio of 1:20.
  • Each node type uses a predefined set of evidence edge types for node-level evidence retrieval.

B.1 GNN Implementation Details and Hyperparameter Selection

The implementation trains relation-specific GNN backbones on leakage-controlled MatKG subgraphs and calibrates candidate decisions on validation data. LLM inference uses shared decoding settings, structured evidence modes, and a fixed JSON review contract supported by naturalization templates.

  • GNN training: Each GNN is trained separately for a target relation after removing validation and test positive edges from the training graph.Edge-support counts are log-transformed and min–max normalized when used as edge weights.
  • GNN training: Validation-selected thresholds maximize F1 before fixed test evaluation, with each candidate receiving a score sh ∈[0, 1].The experiments use Adam with learning rate 10^-3 and random seed 42.
  • GNN architectures: The evaluated backbones include HeteroConv, HGT, and R-GCN variants with relation-aware message passing and distinct embedding-based link decoders.HeteroConv and HGT use normalized endpoint dot products, while R-GCN uses a richer sparse-link edge scorer.
  • LLM inference: Every model emits single-line JSON containing reasoning, a yes/no review, and confidence, with review='yes' iff confidence ≥0.5.Confidence is defined as the probability that the hypothesis is true.
  • LLM inference: The inference sweep spans 7 relations, 4 models, and 3 evidence modes: no-evidence, template-evidence, and LLM-evidence.Template naturalization deterministically converts structured graph evidence into natural-language statements, while LLM evidence uses KG-derived evidence vocabulary.
  • Evidence naturalization: Naturalization templates encode association strength, mechanistic path strength, absent paths, convergent mechanisms, and contradictions for downstream validation.The evidence-writing prompt converts structured KG evidence into a coherent natural-language summary for hypothesis validation.
Loading 2609.02056v1…