Source-linked AI summary
Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation
Junde Wu, Jiayuan Zhu, Yunli Qi, Jingkun Chen, Min Xu, Filippo Menolascina, Vicente Grau
TL;DR
Medical LLMs need reliable reasoning over large, specialized knowledge bases, while existing GraphRAG systems lack medical evidence grounding and can be costly to apply. MedGraphRAG combines Triple Graph Construction with U-Retrieval to connect user data with medical sources and definitions and refine retrieval. It outperformed prior retrieval methods and many medical LLMs across benchmarks, with evaluations supporting more reliable, understandable, and evidence-based responses.
Problem
Medical LLMs require accurate, verifiable reasoning over extensive specialized knowledge, while existing GraphRAG lacks evidence-based response support and is costly to adapt medically.
Method
MedGraphRAG combines Triple Graph Construction linking user data to medical sources and definitions with U-Retrieval’s top-down indexing and bottom-up response refinement.
Results
MedGraphRAG outperformed classic RAG, GraphRAG, and many medically trained LLMs across medical benchmarks, while evaluations supported more reliable and evidence-based responses.
Takeaways & Limitations
The framework provides medical responses with source documentation and definitions while supporting holistic retrieval and response generation.
Abstract
from arXiv · showhide
We introduce a novel graph-based Retrieval-Augmented Generation (RAG) framework specifically designed for the medical domain, called \textbf{MedGraphRAG}, aimed at enhancing Large Language Model (LLM) capabilities for generating evidence-based medical responses, thereby improving safety and reliability when handling private medical data. Graph-based RAG (GraphRAG) leverages LLMs to organize RAG data into graphs, showing strong potential for gaining holistic insights from long-form documents. However, its standard implementation is overly complex for general use and lacks the ability to generate evidence-based responses, limiting its effectiveness in the medical field. To extend the capabilities of GraphRAG to the medical domain, we propose unique Triple Graph Construction and U-Retrieval techniques over it. In our graph construction, we create a triple-linked structure that connects user documents to credible medical sources and controlled vocabularies. In the retrieval process, we propose U-Retrieval which combines Top-down Precise Retrieval with Bottom-up Response Refinement to balance global context awareness with precise indexing. These effort enable both source information retrieval and comprehensive response generation. Our approach is validated on 9 medical Q\&A benchmarks, 2 health fact-checking benchmarks, and one collected dataset testing long-form generation. The results show that MedGraphRAG consistently outperforms state-of-the-art models across all benchmarks, while also ensuring that responses include credible source documentation and definitions. Our code is released at: https://github.com/MedicineToken/Medical-Graph-RAG.
1 Introduction
Medical LLMs need to reason over extensive external knowledge while producing accurate, credible, and verifiable responses. MedGraphRAG addresses GraphRAG’s medical-domain limitations with Triple Graph Construction and U-Retrieval, and reports stronger benchmark and human-evaluation results.
- Motivation: Medical LLMs must handle vast knowledge, precise terminology, and established medical facts without distorting information.The paper emphasizes that non-experts also face difficulty verifying medical response accuracy.
- Limitations of existing approaches: RAG supports private-dataset question answering without additional training, but struggles with holistic synthesis across extensive documents.GraphRAG improves complex reasoning through conceptual relationships, yet its medical application requires additional engineering.
- Method: MedGraphRAG links user RAG data to credible medical sources and foundational dictionaries through Triple Graph Construction, producing traceable sources and definitions.Its graph represents triples of RAG data, source, and definition.
- Method: U-Retrieval uses top-down tag-based indexing followed by bottom-up integration of higher-level tags to refine responses while avoiding costly graph-community construction.The process first identifies a relevant graph for an initial response, then progressively incorporates broader tags.
- Results: Across 9 medical Q&A benchmarks, MedGraphRAG outperformed classic RAG, GraphRAG, and many medically trained LLMs, establishing new state-of-the-art results.The reported comparison spans multiple open-source and commercial LLMs.
- Results: Health fact-checking and clinician evaluations supported MedGraphRAG’s more source-based, reliable, understandable, and evidence-based responses.The paper reports validation on 2 health fact-checking benchmarks and through human evaluation.
2 Method
MedGraphRAG builds hierarchical medical graphs that connect user-document entities to credible sources and definitions, then retrieves information through top-down indexing and bottom-up response refinement.
- Overall workflow: The workflow constructs knowledge graphs from documents, organizes and summarizes them for retrieval, and finally retrieves data to answer user queries.Document processing includes chunking, graph construction, graph organization, and query response.
- Document chunking: Documents are segmented into topic-consistent chunks using paragraph boundaries, a five-paragraph sliding window, semantic consistency, and an LLM-token limit.The hybrid procedure combines character-based separation with topic-based semantic segmentation.
- Triple Graph Construction: The method extracts entities and relationships from each chunk to create directed Meta-MedGraphs whose nodes encode names, types, contexts, and inter-entity relations.Relationships are generated from entity content and associated source references.
- Triple Graph Construction: Triple Graph Construction links RAG entities to medical papers or books and authoritative definitions, yielding triples of RAG entity, source, and definition.Entity links use content-embedding similarity, while the three graph tiers are hierarchically connected through semantic relationships.
- Graph organization: Meta-MedGraphs are summarized with medical tags and clustered by tag similarity into up to 12 increasingly abstract layers without directly linking nodes across graphs.Important tag categories include Symptoms, Patient History, Body Functions, and Medication; dynamic thresholding merges highly similar groups.
- U-Retrieval: U-Retrieval selects a relevant graph through top-down tag matching, retrieves similar entities and triple neighbors, then refines the answer bottom-up using higher-level summaries.In practice, the system retrieves 4-6 layers, combining precise indexing with broader graph context.
3.1 Dataset
The evaluation combines private-style electronic health-record retrieval data, repository resources for sources and vocabulary definitions, and benchmark datasets covering biomedical questions and health fact verification.
- Retrieval data: The study uses MIMIC-IV as retrieval-augmented data representing frequently updated private medical data such as patient electronic health records.MIMIC-IV is described as a publicly available electronic health-record dataset.
- Repository data: Repository data include 4.8 million biomedical academic papers, 30,000 textbooks, health fact-checking publications, and a UMLS graph of authoritative vocabularies and semantic relationships.The repository supports credible sources and authoritative vocabulary definitions.
- Evaluation datasets: The test set contains nine biomedical multiple-choice datasets, two public-health fact-verification datasets, and the collected DiverseHealth test set.The biomedical datasets come from the MultiMedQA suite, while DiverseHealth focuses on health equity.
3.2 Experiment Setting
The experiments compare RAG methods across six language models using shared RAG and test data. MedGraphRAG is illustrated as producing more detailed, evidence-supported explanations than GraphRAG.
- The evaluation compares RAG, GraphRAG, and MedGraphRAG across Llama2, Llama3, Gemini-pro, and GPT-4 models.The Llama variants include 13B, 70B, 8B, and 70B models.
- All retrieval methods use the same RAG data and test data for comparison.
- Figure 2 contrasts GraphRAG and MedGraphRAG responses with their abstracted graphs.
- MedGraphRAG provides more detailed explanations and more complex reasoning with evidences.
3.3 Results
MedGraphRAG improves retrieval-based medical question answering and long-form generation across benchmark and human evaluations. It achieves state-of-the-art results while receiving stronger ratings for citation quality and understandability.
- Nearly 10% average improvement in fact-checking and 8% in medical Q&A occurs versus baselines without retrieval.
- Around 8% average improvement in fact-checking and 5% in medical Q&A occurs versus GraphRAG baselines.
- MedGraphRAG establishes new SOTA results across all 11 datasets when applied to larger models such as Llama70B or GPT.
- MedGraphRAG also surpasses intensively fine-tuned medical models including Med-PaLM 2 and Med-Gemini on the medical LLM leaderboard.
- Across five human-evaluation dimensions, MedGraphRAG consistently receives higher ratings than citation-backed comparison systems.The dimensions are pertinence, correctness, citation precision, citation recall, and understandability.
- Its strongest reported advantages are citation precision, citation recall, and understandability, reflecting accurate sources and clearer medical-term explanations.
3.4 Analysis
The analysis separates the effects of repository data from retrieval design. MedGraphRAG benefits most from combining additional medical data with its hierarchical retrieval method.
- The experiments indicate that repository data and retrieval method must work together to unlock the data’s full potential.
- Standard RAG gains less than 2% from Med-Paper data and less than 1% from Med-Dictionary data individually.
- GraphRAG improves with individual data retrieval but shows minimal gains when additional data is added incrementally.
- MedGraphRAG shows strong improvements as more data is added through its hierarchical structure.
- Over 6% improvement occurs with Med-Paper data and over 8% with Med-Dictionary data under MedGraphRAG.
4 Related Work
Prior work includes specialized medical LLMs and cost-efficient non-fine-tuning approaches. The paper positions medical RAG as underexplored and lacking evidence-based responses and medical-term explanations.
- Specialized medical LLMs include BioGPT, PMC-LLaMA, BioMedLM, and Med-PaLM 2.
- Recent research also explores cost-efficient approaches based primarily on prompt engineering rather than fine-tuning.
- Medical applications have rarely explored RAG as a non-fine-tuning approach.
- Existing medical RAG work lacks support for evidence-based responses and term explanations required in clinical settings.
- RAG enables models to use specific datasets without additional training and has been applied to citation-generating responses.
5 Conclusion
MedGraphRAG improves the reliability of medical response generation through Triple Graph Construction and U-Retrieval. Its benchmark and human-evaluation performance supports accuracy in complex medical reasoning.
- MedGraphRAG uses Triple Graph Construction and U-Retrieval to produce evidence-based, context-aware medical responses.
- Benchmark and human evaluations show strong performance for accuracy in complex medical reasoning.
- Future work will address real-time data updates and validation on real-world clinical data.