Source-linked AI summary
Graph-Based Retriever Captures the Long Tail of Biomedical Knowledge
Julien Delile, Srayanta Mukherjee, Anton Van Pamel, Leonid Zhukov
TL;DR
Biomedical literature overload makes rare and recent findings difficult to retrieve amid over-represented concepts. The paper uses a biomedical knowledge graph to rebalance retrieval and combines it with embedding similarity; the methods' complementarity supports a hybrid that outperforms either individually.
Problem
Biomedical literature contains redundant information, so rare and recent findings can be dominated by over-represented concepts.
Source: §1 · p. 2Method
The approach maps biomedical text chunks to graph entities and relations, retrieves along paths between question entities, and ranks chunks using recency and document citation impact.
Source: §3.2 · p. 4, §3.2.3 · p. 5Results
Takeaways & Limitations
Graph-based balancing and embedding similarity are complementary, with graph retrieval identifying a more diverse set of relevant documents.
Source: §5 · p. 8The study does not test how access to biomedical long-tail information affects answer generation.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Large language models (LLMs) are transforming the way information is retrieved with vast amounts of knowledge being summarized and presented via natural language conversations. Yet, LLMs are prone to highlight the most frequently seen pieces of information from the training set and to neglect the rare ones. In the field of biomedical research, latest discoveries are key to academic and industrial actors and are obscured by the abundance of an ever-increasing literature corpus (the information overload problem). Surfacing new associations between biomedical entities, e.g., drugs, genes, diseases, with LLMs becomes a challenge of capturing the long-tail knowledge of the biomedical scientific production. To overcome this challenge, Retrieval Augmented Generation (RAG) has been proposed to alleviate some of the shortcomings of LLMs by augmenting the prompts with context retrieved from external datasets. RAG methods typically select the context via maximum similarity search over text embeddings. In this study, we show that RAG methods leave out a significant proportion of relevant information due to clusters of over-represented concepts in the biomedical literature. We introduce a novel information-retrieval method that leverages a knowledge graph to downsample these clusters and mitigate the information overload problem. Its retrieval performance is about twice better than embedding similarity alternatives on both precision and recall. Finally, we demonstrate that both embedding similarity and knowledge graph retrieval methods can be advantageously combined into a hybrid model that outperforms both, enabling potential improvements to biomedical question-answering models.
1 Introduction
Biomedical literature is growing too rapidly for individuals to track, while information overload makes retrieval of rare and recent findings difficult. The paper proposes graph-based retrieval and a hybrid approach to address this challenge.
1 Introduction
- Biomedical research spans specialized disciplines and an expanding literature, making it increasingly difficult for individuals to track its evolving landscape.The passages cite over 42 million unique genes indexed by NCBI as one indication of this scale.Source: §1 · p. 1
- RAG combines an external knowledge source with an LLM to retrieve and synthesize information, without requiring model retraining for updates.The introduction describes RAG as an alternative to storing information in model weights through training or fine-tuning.Source: §1 · p. 1
- Biomedical literature presents an information-overload problem: rare, recent, important information can be dominated by over-represented older concepts.The passages also note that longer retrieved contexts can be harder for LLMs to exploit efficiently.Source: §1 · p. 2
- The paper proposes graph-based retrieval to rebalance retrieved text chunks and a hybrid combining graph structure with embedding semantics that outperforms both individual methods.The proposed approach also includes controls to prioritize recent and impactful discoveries.Source: §1 · p. 2
2 Related work
Prior work combines knowledge-graph paths and language-model encodings for multiple-choice QA, while a KG index retrieves nearby triplets for answer generation. This study addresses information overload in embedding-similarity retrieval with graph-distance retrieval.
2 Related work
- Several multiple-choice QA models combine question-answer encodings with graph embeddings of KG paths, but they do not generate natural-language responses.Examples include KagNet, MH-GRN, QA-GNN, JointLK, and GreaseLM.Source: §2 · p. 2
- KGLM uses LSTMs to predict graph entities as tokens, while the LlamaIndex KG index retrieves triplets from keyword-centered k-hop neighborhoods for answer generation.The retrieved triplets are added to the synthesizer context.Source: §2 · p. 2
- The study presents graph-distance retrieval as a response to information overload in embedding-similarity retrieval.The authors describe it as, to their knowledge, the first such highlighting of the problem and proposal of this approach.Source: §2 · p. 2
3 Methods
The methods compare embedding-similarity retrieval with a biomedical knowledge-graph approach. The graph method maps annotated text chunks to entities and relations, retrieves along shortest paths between question entities, and ranks results to balance access across entities while surfacing recent, significant findings.
3 Methods
- The methods compare dense-embedding similarity retrieval with a knowledge-graph approach based on entity recognition and relationship extraction.The retrieval step selects text chunks; synthesis combines retrieved context and the question to produce an answer.Source: §3 · p. 3
3.1.1 Embedding indexing
- The embedding index uses article titles and abstracts from PubMed, split into sentences and embedded with text-embedding-ada-002.The indexed articles are a subset of those with abstracts; experiments typically index about 100,000 documents.Source: §3.1.1 · p. 3
3.1.2 Retrieval
- Embedding-similarity retrieval ranks text chunks by cosine similarity between their embeddings and the user-question embedding.The study uses cosine similarity and varies retrieval-rank thresholds across experiments.Source: §3.1 · p. 3, §3.1.2 · p. 4
3.2 IR with knowledge graph support
- The graph approach uses biomedical entities to define information clusters that can be undersampled to rebalance retrieval.The proposed clusters are built around genes, diseases, and chemical compounds.Source: §3.2 · p. 4
3.2.1 Building the biomedical Knowledge Graph
- The biomedical graph combines KAZU entity extraction and normalization with PubmedBERT relationship extraction fine-tuned on BioRED.Predicted relationships include association, positive or negative correlation, cotreatment, comparison, and binding.Source: §3.2.1 · p. 4
3.2.2 Knowledge Graph Indexing
- Text chunks with one entity map to its node; chunks with semantically linked entity pairs map to an edge, while unlinked pairs map to entity nodes.The rules also apply to all combinatorial entity pairs in chunks containing three or more entities.Source: §3.2.2 · p. 4
3.2.3 Retrieval
- Graph retrieval identifies question entities, retrieves chunks mapped to their shortest path and neighboring edges, then ranks results to balance entities and surface recent, significant discoveries.The ranking metric factors in recency and document citation counts, using their Pareto front to address the trade-off.Source: §3.2.3 · pp. 4–5
- Graph retrieval can return chunks along paths between question entities, but hundreds of thousands may map to a path, motivating prioritization.Optional publication-year and publication-count thresholds are not used in this study to keep the comparison fair.Source: §3.2.3 · pp. 4–5
4 Experiments
Across eight disease questions, knowledge-graph retrieval outperformed embedding similarity on the reported retrieval metrics, while a hybrid method performed best at smaller retrieval volumes. The results link KG retrieval’s broader coverage to balancing over-represented concepts and show that combining it with semantic similarity can improve retrieval.
4.1 Comparing knowledge graph and embedding similarity retrieval performance
- The comparison used expert-curated annotations for an open drug-target question across eight diseases, with a shared corpus drawn from five therapeutic areas.The authors note that incomplete annotations may underestimate both methods’ performance, but assume this does not favor either approach.Source: §4.1 · pp. 5–6
- At K=250, KG IR retrieved nearly 43% of gold-standard documents versus 17% for ES IR, although ES IR recall exceeded KG IR at K>1000.KG IR retrieves only documents mapped to the knowledge graph, whereas ES IR searches the full embedded corpus.Source: Figure 3 · p. 6, §4.1 · p. 7
4.2 Knowledge graph retrieval accesses the long tail of biomedical knowledge
- At equal retrieval volume, ES IR covered less than half as many embedding-space clusters as KG IR, whose retrieved regions also included smaller clusters of curated documents.The authors attribute ES IR’s concentration near the question embedding to a lack of data balancing.Source: §4.2 · pp. 7–8
- KG IR’s balancing mechanism reaches relevant information beyond the question’s immediate embedding neighborhood, supporting retrieval of long-tail biomedical knowledge.The projected landscape showed localized disease areas and distinct regions associated with different entity types.Source: §4.2 · pp. 7–8
4.3 Semantic and graph-based retrieval approaches are highly complementary
- Hybrid IR averages independently rescaled embedding-similarity and KG-priority scores, combining semantic filtering with graph-based data rebalancing.Only text chunks mapped to the knowledge graph are eligible for hybrid retrieval.Source: §4.3 · p. 8
- At K=50, Hybrid IR had about twice the recall and precision of KG IR, and it outperformed both methods at K<100 and moderately beyond K=250.Hybrid IR averages each method’s independently rescaled scores; only text chunks mapped to the knowledge graph are available to it.Source: Figure 3 · p. 6, §4.3 · p. 8
5 Discussion
The discussion argues that KG retrieval reaches beyond the immediate question neighborhood and complements embedding similarity, making their hybrid superior to either method alone. It also notes that the study did not test whether access to long-tail biomedical information improves answer generation.
5 Discussion
- KG retrieval extends beyond the question embedding’s immediate neighborhood, while its complementarity with embedding similarity supports a hybrid that outperforms either method alone.The authors link ES IR’s oversampling of nearby content to its lack of data balancing.Source: §5 · p. 8
- The study did not test whether retrieving biomedical long-tail information improves answer generation, and the authors flag a possible signal-to-noise trade-off.They describe KG IR precision as low, though still better than ES IR, and suggest controls may help maintain performance.Source: §5 · p. 8
- The authors propose chain-of-thought-guided retrieval as future work, particularly for complex questions requiring multi-hop retrieval or subgraph identification.They suggest it could help direct entity mapping and declutter neighboring entities that may be irrelevant.Source: §5 · p. 8
- They also propose reversing the RAG setup to generate text from relevant graph entities, building on a weakly supervised, domain-specific graph-text corpus.Potential applications named include scientific-article authoring and regulatory submissions.Source: §5 · p. 10