Assessing LLMs for Serendipity Discovery in Knowledge Graphs: A Case for Drug Repurposing

1 Introduction

• Existing LLM systems can return relevant, precise answers while favoring information familiar to experts over surprising connections that may inspire research.The paper frames uncovering valuable new knowledge from knowledge bases as a step toward LLM-enabled scientific discovery.Source: §1 · p. 1
• Journavx illustrates a serendipitous answer for severe acute pain: its novel NaV1.8 mechanism remains relevant through a broader pain-signal context shared with opioids.The example contrasts it with familiar opioids and Ibuprofen, which the passage describes as low in embedding relevance and novelty for severe acute pain.Source: §1 · pp. 1–2
• SerenQA evaluates scientific KGQA serendipity through an RNS metric, an expert-annotated drug-repurposing benchmark, and a three-phase assessment pipeline.RNS measures relevance, novelty, and surpriseness; the pipeline evaluates knowledge retrieval, reasoning, and exploratory search.Source: §1 · p. 2

LLM-Augmented Novelty Detection

• The paper presents a drug-repurposing KGQA dataset to assess serendipitous knowledge exploration amid a lack of comprehensive understanding and dedicated benchmarks.It describes SerenQA as the first reproducible and extensible framework for advancing serendipity discovery in drug repurposing.Source: LLM-Augmented Novelty Detection · p. 2

Serendipity-aware KGQA

• Serendipity-aware KGQA returns an ordered partition of explicit answers Ae and relevant, novel answers As that extend beyond direct graph knowledge.Ae and As are disjoint subsets of graph entities; together they form the total answer set.Source: Serendipity-aware KGQA · p. 2

2.2 Graph-specified Serendipity Formulation

• The RNS score combines relative relevance, novelty, and surprise to quantify the serendipity of As beyond explicit answers Ae.The metric’s weights can be tuned to user preferences, with recommended defaults fit to expert evaluations.Source: §2.2 · pp. 2–3
• The formulation supports a unified benchmark, dataset, and evaluation protocols for assessing LLMs in serendipitous knowledge discovery, particularly drug repurposing.Source: §2.2 · p. 3

3.1 Serendipity: A characterization

• SerenQA defines RNS as a user-weighted combination of relative relevance, novelty, and surprise between existing answers Ae and serendipity answers As.User-configurable weights allow different preferences or application scenarios.Source: §3.1 · p. 3
• Relative relevance uses GCN-embedding distances, while novelty is 1 − MI(Ae, As), measuring how much As is nonredundant given Ae.A larger embedding distance reflects greater contextual difference between answer sets.Source: §3.1 · p. 3
• Relative surprise is measured with Jensen–Shannon divergence between the entity distributions Ps and Pe of As and Ae, respectively.Source: §3.1 · p. 3

3.2 Cost-effective Graph Probabilistic Modeling

• 99% of serendipity answers in the datasets are reachable from existing answers within three hops, motivating a multi-hop transition model; damped iteration computes marginals in O(V^2 log V) time.The probability matrices are computed once and shared across queries, and can adapt to different domain graphs.Source: §3.2 · p. 4
• RNS satisfies scale invariance, consistency, non-monotonicity, and independence, supporting a semiclosed-world measure based only on embeddings of entities in As ∪ Ae.Larger answer sets need not be more serendipitous, and unseen entities do not affect RNS.Source: §3.2 · p. 4

4 Serendipity-aware Benchmark

• The benchmark extends the Clinical Knowledge Graph with drug-repurposing questions and serendipity ground truth for standard and serendipity-aware KGQA evaluation.Drug repurposing targets novel therapeutic uses for existing drugs.Source: §4 · p. 4, §4.1 · p. 4

4.1 QA Set Construction

• Each expert-verified natural-language query is paired with a semantically annotated Cypher query and grounded, validated answers, reducing ambiguity and capturing multi-hop and intersection patterns.This contextualized query scenario links each natural-language query to a distinct structured graph query, unlike NL-only KGQA datasets.Source: §4.1 · p. 4

4.2 Answer Set Construction

• For each query, the complete candidate set is divided into existing and serendipity answers; four LLMs rank candidates, and experts refine the rankings.The LLM ensemble assigns scores whose averages select the top 20% as serendipity answers; the expert panel comprises six domain experts, including physicians and a pharmaceutical scientist.Source: §4.2 · pp. 4–5
• The RNS-guided strategy approximately optimizes the serendipity partition with fixed size b = max(1, ⌊0.2|Ac|⌋), using greedy swaps and weights calibrated to expert-crowdsourced partitions.Each iteration has complexity O(|A|^2); existing-answer sets average four entities, and the algorithm is reported to be fast in practice.Source: §4.2 · p. 5
• For each partition, selected edges are removed from the Clinical Knowledge Graph so existing answers remain derivable while serendipity answers become inaccessible.This creates a controlled evaluation environment aligned with the task definition.Source: §4.2 · p. 5

Knowledge Retrieval

• Knowledge retrieval translates natural-language questions into Cypher queries and scores retrieved answers against ground truth across query patterns of varying complexity.The patterns include one-hop, two-hop, and intersection queries.Source: §5 · p. 5

Subgraph Reasoning

• Subgraph reasoning summarizes retrieved graph-query answers in domain-aware natural language, providing biomedical context for later serendipity exploration.The task requires nuanced biomedical understanding and logical reasoning.Source: §5 · p. 5

Serendipity Exploration

• Serendipity exploration uses LLM-guided beam search from retrieved answers to select candidate entities and decide whether to continue based on relevance and potential novelty.Selection considers supporting evidence, interaction strength, biological effect direction, and expression level; the evaluated output set is distinct from the benchmark ground-truth set used for scoring.Source: §5 · p. 5

6.1 Experiment Setting

• The experiments span frontier to smaller LLMs and evaluate knowledge retrieval, subgraph reasoning, and serendipity exploration using one-shot path guidance during beam search.Beam search uses width 30 and maximum depth 3. Metrics include retrieval F1, executability, and hit rate; reasoning Faithfulness, Compre., and SerenCov; and exploration Relevance, TypeMatch, and SerenHit.Source: §6.1 · pp. 5–6

6.2 Task Analysis

• F1 is approximately 78% for larger models on one-hop retrieval but drops below 10% on queries with 3+ hops; the two 70B models perform better than smaller models.The reported larger models include DeepSeek-V3 and GPT-4o; the 70B models are Llama-3.3-70B and DeepSeek-R1-70B.Source: Table 2 · p. 6, §6.2 · p. 6

6.3 In-Depth Discussion

• Larger models generally perform better in retrieval and exploration, but subgraph reasoning shows significant variance and no clear relationship with model size.The authors suggest retrieval and exploration may benefit more from larger models’ inherent knowledge than summarization and reasoning do.Source: §6.3 · p. 6

6.2 Task Analysis

• Mixtral-8×7B exceeds 60% SerenCov despite Faithfulness and Compre. scores of 2–3/5, while Llama-3.3-70B favors higher faithfulness and comprehensiveness over coverage.The results suggest that broader serendipitous-path coverage can come with factual-accuracy risks, whereas richer summaries may be more precise but narrower.Source: §6.2 · p. 6, Table 3 · p. 7
• For almost all models, removing subgraph summaries improves all three serendipity-exploration metrics; the authors suggest summary hallucinations may influence exploration paths.The comparison is between the “w.o. summary” rows and results with subgraph summaries.Source: §6.2 · p. 6, Table 3 · p. 7

6.3 In-Depth Discussion

• All partitions correlate positively across metrics at above 85%; expert and RNS-guided partitions reach around 99% in all cases.The authors interpret these results as evidence of partition-strategy robustness and RNS-measure reliability.Source: §6.3 · pp. 6–7, Figure 3 · p. 7
• No model leads every metric: DeepSeek-R1-70B excels in retrieval but is moderate in reasoning and poor in exploration, while Llama-3.3-70B is more versatile; multi-model strategies may help.The suggested approaches include multi-agent systems and mixtures of experts.Source: §6.3 · p. 7

• SerenQA combines an axiomatically justified measure of relevance, novelty, and surprise with a serendipity-aware drug-repurposing benchmark.Source: §7 · p. 7
• Its evaluation pipeline assesses knowledge retrieval, subgraph reasoning, and serendipity exploration.Source: §7 · p. 7
• Frontier LLMs excel at basic knowledge retrieval but often struggle with complex queries and answers for serendipity exploration.The findings indicate substantial room for improvement.Source: §7 · p. 7

Ethical Statement

• The approach does not account for druggability factors such as physicochemical properties, and the clinical effectiveness of suggested associations remains uncertain pending preclinical and clinical validation.The study used LLMs to identify serendipitous drug–disease associations that may suggest novel therapies.Source: Ethical Statement · p. 8

A. Dataset Details

• The dataset details section includes a subsection on dataset construction.Source: A. Dataset Details · p. 9
• The dataset details section includes a subsection on pattern type.Source: A. Dataset Details · p. 9
• The dataset details section includes a subsection on dataset structure.Source: A. Dataset Details · p. 9
• The dataset details section includes a subsection presenting more statistics.Source: A. Dataset Details · p. 9

B. Prompts -

• The prompts section includes LLM scoring prompts.Source: B. Prompts - · p. 9
• The prompts section includes serendipity exploration prompts.Source: B. Prompts - · p. 9
• The prompts section includes pipeline evaluation prompts.Source: B. Prompts - · p. 9

C. Further Analysis on RNS Metric

• The RNS metric analysis is divided into k-hop conditional probability matrix and marginal probability subsections.Source: C. Further Analysis on RNS Metric · p. 9

D. Details of Serensipity Exploration

• Serendipity exploration details cover workflow and logic, infrastructure, and neighbor scoring.Source: D. Details of Serensipity Exploration · p. 9

E. Experiment Details

• The experiment details section includes experiment setting and additional analysis.Source: E. Experiment Details · p. 9

A.1 Dataset Construction

• The CKG contains ∼20 million nodes across 36 types and over 220 million edges across 47 relationship types, providing the biomedical graph foundation.It includes entities such as genes, proteins, diseases, and drugs, linked by relationships including drug indications and drug-target interactions.Source: §A.1 · p. 9, Figure 4 · p. 10
• The QA dataset uses a CKG subgraph that excludes user-, unit-, experiment-, project-, transcript-, and publication-related node and edge types.Source: §A.1 · p. 9
• The current dataset has 1,529 drug-disease-focused queries annotated with relevant nodes, edges, a target node, and graph metadata; expansion to 5,000 queries is planned.The planned expansion is intended to support broader NLP tasks and more comprehensive evaluation of biomedical reasoning.Source: §A.1 · p. 9
• One- and two-hop answers were queried directly in Neo4j, while three-hop and intersection neighborhoods were pre-extracted; question templates were refined for clear, biologically relevant oncology phrasing.Source: §A.1 · p. 9

A.2 Pattern Type

• One-hop questions retrieve entities linked by a single relationship, either from a specified source or by identifying sources connected to a specified target.Two-hop questions link a source to a final entity through an intermediate entity and two relationships.Source: §A.2 · p. 10
• Three-hop questions trace three-edge paths through two intermediate entities, including sequential, hierarchical, and branching connection patterns.Source: §A.2 · p. 11
• Intersection questions identify entities sharing connections across paths, ranging from basic and multi-way overlaps to compound cyclic patterns.Source: §A.2 · p. 11

A.3 Dataset Structure

• Standard KGQA records pair expert-verified natural-language queries and structured Cypher queries with validated answers, while serendipity records partition graph candidates and provide ground-truth paths between sets.The serendipity-aware format also includes exploration queries and partition information.Source: §A.3 · pp. 11–12

A.4 More Statistics

• The KGQA dataset contains 1,529 drug-repurposing queries, with Table 4 presenting their distribution across question-pattern types.Source: §A.4 · p. 14, Table 4 · p. 14

B.1 LLM Scoring Prompts

• The evaluator prompt scores each correct answer from 0–20 for relative novelty and insight, assigns distinct scores, and evaluates answers independently of list position.A score of 20 represents a highly novel, unexpected, or insightful answer; 0 denotes a correct but obvious answer with no novelty.Source: §B.1 · p. 14

B.2.1 Select Relation System Prompt:

• The relation-selection prompt asks for top-m predicates to build three-hop paths that balance biological plausibility and mechanistic richness with novel, potentially serendipitous exploration.It prioritizes biologically meaningful sequences while balancing frequent paths against rare combinations.Source: §B.2.1 · p. 15

B.2.2 Select Nodes

• The node-selection prompt continues from selected relations to target nodes, aiming to construct three-hop paths that are biologically plausible, serendipitous, and mechanistically rich.Source: §B.2.2 · p. 16
• Node selection prioritizes 3-hop biological-mechanism paths, balancing high-frequency plausible paths with rare combinations and avoiding trivial paths.The listed relation categories include curated/compiled, functional/structural, annotation, and rare/high-value relations.Source: §B.2.2 · p. 16

B.2.3 Decide Whether to Continue

• Before expanding, the system reviews the root-to-current 3-hop path, summarizes its biological context, and judges whether further exploration remains justified.Continuation depends on whether the current node plausibly supports mechanistic or therapeutic insight relevant to the original question.Source: §B.2.3 · p. 17

B.2.4 Summarize Subgraph System Prompt:

• Subgraph summaries cover retrieved 1-, 2-, and 3-hop neighborhoods, emphasizing biomedical terms, entity and relation patterns, and biological or therapeutic meaning.The requested summary is concise and focuses on mechanistic implications or potential therapeutic relevance of paths.Source: §B.2.4 · p. 17

B.3.1 Faithfulness Assessment System Prompt:

• Assessment separates exact-match retrieval for entities directly answering the question from serendipity exploration expanding up to 3 hops from root nodes to propose new entities; faithfulness asks whether summaries truthfully reflect graph contents without inventing entities.Source: §B.3.1 · p. 18

B.3.2 Comprehensiveness Assessment

• The comprehensiveness rubric holistically assesses whether hop summaries cover important graph entities, relations, and mechanistic chains, without scoring by token length.Higher scores indicate that nearly all salient triples or concepts appear; lower scores reflect missing key relationships, nodes, or structure.Source: §B.3.2 · p. 20

B.3.3 Relevance Assessment

• Relevance is judged against an expert-curated ground-truth serendipity set by functional or clinical relatedness, not entity-type overlap alone.Minor naming variants or isoforms are acceptable.Source: §B.3.3 · p. 21

Properties Verification

• The weighted multi-hop conditional probability matrix Pk is non-negative and row-stochastic, so it remains a valid transition probability matrix.The proof uses non-negative powers of the row-normalized base matrix and positive hop-weight coefficients, then verifies row sums equal one.Source: Properties Verification · p. 22

Computation

• Strassen’s divide-and-conquer multiplication reduces Pk computation from O(V^3) to approximately O(V^2.807), or O(V^2.807/t) with t processors.The method recursively divides matrices into four submatrices and reduces multiplications per recursion from eight to seven.Source: Computation · p. 22

C.2 Marginal Probability

• Marginal probability computation is approximated with a PageRank-style damped iteration.The passage refers to Algorithm 2 but does not specify further iteration details.Source: §C.2 · p. 22
• The marginal probability vector is iteratively updated with damping until tolerance is met, with total convergence complexity O(V^2 log V).Each iteration is dominated by matrix-vector multiplication at O(V^2).Source: §C.2 · p. 23

D.1 Workflow and Logic

• A multi-stage beam search recursively explores root-node neighborhoods, using LLM-guided expansion and multiple LLM interactions for external reasoning.Source: §D.1 · p. 23

D.2 Infrastructure

• Redis-based edge storage raised query throughput from 1,000 QPS to tens of thousands, supporting concurrency of approximately 100 per compute node.The AWS cluster used five compute nodes and five Redis storage nodes; each compute node had 96 vCPUs and 768 GiB of memory.Source: §D.2 · p. 23

D.3 Neighbor Scoring

• Neighbor scoring reduces LLM input at high-degree hubs by using extracted confidence or domain-specific scores, including normalized, aggregated expression and prognosis signals.Expression scores reflect categorical levels, while prognostic scores use log-transformed p-values for positive or negative survival associations.Source: §D.3 · p. 23

• Evaluation covered frontier, ~70B, 20–50B, and small language-model cohorts.The models were DeepSeek-V3, GPT-4o, Claude-3.5-Haiku; Llama-3.3-70B, DeepSeek-R1-70B, Qwen-2.5-72B, Med42-V2-70B; Mixtral-8x7B, Qwen3-32B, DeepSeek-R1-32B, Gamma-2-27B, Mistral-24B; and Qwen-2.5-7B, Qwen3-8B, DeepSeek-R1-8B, Med42-V2-8B, Qwen3-1.7B, DeepSeek-R1-1.5B.Source: §E.1 · p. 24

Subgraph Reasoning

• Subgraph reasoning averages rational-sample scores for LLM-judged faithfulness and comprehensiveness plus code-based serendipity coverage.Faithfulness and comprehensiveness are scored 1–5; coverage is the 0–1 fraction of serendipity paths whose source and target node IDs both appear explicitly in the summary, checked by regex matching.Source: §E.1 · p. 24

Serendipity Exploration

• Serendipity exploration averages rational-sample scores for relevance, TypeMatch, and SerenHit.Relevance is LLM-judged on a 1–5 scale; code-based TypeMatch (0–1) is 1 if any predicted leaf matches a ground-truth type, while SerenHit (0–1) is 1 if any predicted leaf exactly matches the ground-truth serendipity set.Source: §E.1 · p. 24

Model Scale vs. Serendipity Exploration

• SerenHit remains below 0.10 as model size grows from 7B to 70B, while Relevance improves gradually and TypeMatch and SerenHit increase inconsistently.The analysis concludes that larger parameter counts alone are insufficient for reliable, precise serendipitous discovery.Source: §E.2 · p. 24, Figure 5 · p. 25

Algorithm 3: LLM Enhanced Beam Explore

• LLM Enhanced Beam Explore selects relation types and nodes, scores candidate edges for top-k filtering, and tracks paths from the root across beam levels.When every edge score fails, it uniformly samples candidates; the algorithm can also stop based on an LLM continuation decision and generates summaries by level.Source: §E.2 · p. 26

Multi-Task Performance Compass

• DeepSeek-V3 excels at F1 and Hit but underperforms in Serendipity Coverage and SerenHit; Llama-3-70B scores highly in Faithfulness and Comprehensiveness, while DeepSeek-R1-70B covers serendipity paths at the cost of reasoning accuracy.No model dominates across all metrics; the analysis suggests ensemble or Mixture-of-Experts approaches.Source: §E.2 · p. 24, Figure 6 · p. 27

Query Pattern vs. Retrieval Performance

• All models’ strong one-hop F1 and Hit scores fall sharply on two-hop queries and further on ≥3-hop or intersection queries.The analysis attributes this decline to difficulty with complex multi-step reasoning and domain-specific context, including for the largest frontier models.Source: §E.2 · p. 24, Figure 7 · p. 28