PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledge

Mohammad Reza Rezaei, Reza Saadati Fard, Jayson L. Parker, Rahul G. Krishnan, Milad Lankarany

arXiv:2502.13010v3cs.CLcs.MA

TL;DR

AMG-RAG addresses the challenge of keeping medical QA current as knowledge evolves and manual graph maintenance remains costly. It automatically builds and updates a medical knowledge graph and combines it with retrieval and reasoning; it reports strong results on MEDQA and MEDMCQA, including performance beyond much larger models.

  • Problem

    Medical QA must handle rapidly evolving knowledge, while manually creating and maintaining medical knowledge graphs is costly.

    Source: §1 · p. 1

  • Method

    AMG-RAG uses LLM agents and search to build and refine a medical knowledge graph, then combines graph-conditioned retrieval with reasoning over structured and textual evidence.

    Source: §1 · p. 1, §3 · p. 2, §3.3 · p. 5

  • Results

    AMG-RAG outperforms similarly sized RAG approaches and much larger models, achieving an F1 of 74.1% on MEDQA and an accuracy of 66.34% on MEDMCQA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings highlight the potential of automated, relationally enriched knowledge retrieval to support clinical decision-making with timely and trustworthy insights.

    Source: §1 · p. 2

    Structured access to treatment guidelines remains future work, and applicability beyond medical tasks is unexplored.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have significantly advanced medical question-answering by leveraging extensive clinical data and medical literature. However, the rapid evolution of medical knowledge and the labor-intensive process of manually updating domain-specific resources pose challenges to the reliability of these systems. To address this, we introduce Agentic Medical Graph-RAG (AMG-RAG), a comprehensive framework that automates the construction and continuous updating of medical knowledge graphs, integrates reasoning, and retrieves current external evidence, such as PubMed and WikiSearch. By dynamically linking new findings and complex medical concepts, AMG-RAG not only improves accuracy but also enhances interpretability in medical queries. Evaluations on the MEDQA and MEDMCQA benchmarks demonstrate the effectiveness of AMG-RAG, achieving an F1 score of 74.1 percent on MEDQA and an accuracy of 66.34 percent on MEDMCQA, outperforming both comparable models and those 10 to 100 times larger. Notably, these improvements are achieved without increasing computational overhead, highlighting the critical role of automated knowledge graph generation and external evidence retrieval in delivering up-to-date, trustworthy medical insights.

1 Introduction

Medical QA systems must keep pace with changing knowledge and model relationships among medical entities, while manually maintaining medical knowledge graphs is costly. AMG-RAG automates graph construction and refinement, combining relational retrieval and multi-step reasoning; it reports strong performance on MEDQA and MEDMCQA.

1 Introduction

  • LLMs face the challenge of staying current as medical knowledge changes and of modeling intricate relationships among medical entities.Hand-building and maintaining knowledge graphs is costly, especially when new findings can invalidate older facts.Source: §1 · p. 1
  • 74.1% F1 on MEDQA and 66.34% accuracy on MEDMCQA; AMG-RAG surpasses similarly sized RAG systems and models with 10–100× more parameters.The figure compares performance with parameter count across both benchmarks.Source: Figure 1 · p. 1, §1 · p. 2
  • AMG-RAG combines a dynamically synthesized medical knowledge graph with confidence-guided, adaptive multi-step reasoning, without additional fine-tuning or inference costs.The graph-centric retrieval uses explicit relationships to synthesize information across domains.Source: §1 · p. 2
  • Specialized LLM agents use search engines and medical textbooks to generate and refine medical knowledge graphs continuously.The system attaches metadata, confidence scores, and relevance indicators to graph entities.Source: §1 · pp. 1–2
  • The authors identify confidence scoring and adaptive graph traversal as contributions for representing uncertainty and contextualizing medical knowledge.They describe confidence scores as transparent reliability assessments.Source: §1 · p. 2

2 Related Work

Prior medical QA work spans domain-specific language models, retrieval-augmented generation, and knowledge-graph reasoning. The approaches address different needs, while challenges remain in multi-hop synthesis, freshness, explicit relations, and scalable graph curation.

Related work overview

  • The paper groups medical QA research into domain-specific language models, retrieval-augmented generation, and knowledge-graph reasoning.Source: §2 · p. 2

Domain-specific language models

  • BioBERT, PubMedBERT, and MedPaLM adapt pretraining to biomedical corpora, but specialized models can struggle with multi-hop relations and require retraining to absorb discoveries.The cited examples include rare comorbidities and drug–gene interactions.Source: §2 · p. 2

Retrieval-Augmented Generation (RAG)

  • RAG adds external evidence to LLM generation, and CoT methods can interleave retrieval with step-wise justification, but most systems rely on static vector stores without explicit biomedical relations.The passage also describes biomedical and multimodal RAG variants and reports Gemini’s long-context model exceeding GPT-4 on MedQA.Source: §2 · p. 2

Knowledge-graph reasoning

  • KG-Rank and related approaches use ontologies to re-rank evidence or enforce logical constraints, while high-coverage, up-to-date medical graphs remain labor-intensive to build and curate.Source: §2 · p. 2

3 Method

AMG-RAG couples a dynamically generated, confidence-scored medical knowledge graph with retrieval and chain-of-thought reasoning. The method combines external evidence, graph traversal, and entity-relation construction to support structured medical question answering.

3 Method

  • AMG-RAG dynamically generates a confidence-scored medical knowledge graph from live sources and couples graph-conditioned retrieval with reasoning over text passages and traversed subgraphs.The answer generator uses both forms of context for transparent, multi-hop reasoning.Source: §3 · p. 2

3.1 Retrieval Augmented Generation (RAG)

  • External search combines additional retrieved documents with the RAG retriever’s results, and this variant significantly enhances performance, particularly for queries needing extensive and diverse knowledge.The basic RAG approach passes a fixed number of query-relevant documents to a text generator, but may struggle with complex or domain-specific queries needing more evidence.Source: §3.1 · p. 3

3.2 Medical QA with AMG-RAG

  • For each query, AMG-RAG extracts medical terms, explores graph nodes and relationships using confidence thresholds, and supports breadth-first or depth-first traversal until a confidence threshold or document limit is reached.It synthesizes reasoning traces for entities and aggregates them into an answer with an associated confidence score.Source: §3.2 · pp. 3–4
  • AMG-RAG is presented as combining structured medical knowledge and reasoning to support comprehensive, interpretable answers anchored in reliable medical knowledge and flexibility for new data.Source: §3.2 · pp. 3–4

3.3 Dynamic Generation of the Medical Knowledge Graph

  • The medical knowledge graph stores entities, descriptions, inferred relationships, and confidence metrics; edge reliability scores inform downstream evidence weighting, while the graph can incorporate new information and refine connections.Medical search tools supply entity descriptions, and an LLM agent infers relationships and their reliability from pairs of nodes.Source: §3.3 · pp. 4–5

4 Experiments

Experiments evaluate AMG-RAG on MEDQA and MedMCQA, while describing how its medical knowledge graph is constructed, validated, and updated. Benchmark comparisons test its performance against larger models and alternative retrieval strategies.

4 Experiments

  • The MEDQA test set contains approximately 1,200 questions, while the MedMCQA development set contains approximately 4,000 questions.MEDQA questions derive from medical board exams; MedMCQA covers foundational and clinical knowledge across specialties.Source: §4 · p. 5

4.1 Medical Knowledge Graph

  • The MKG is built from retrieved search items, contextual information, and medical-textbook relationships, with confidence scores used to monitor and filter uncertain connections.The graph includes bidirectional relationships and summaries with reliability indicators; PubMed-MKG outperforms Wiki-MKG in the reported ablation.Source: §4.1 · pp. 5–6
  • Clinical experts and expert LLM evaluators validated the graph, with expert LLMs such as GPT-4 reaching 9/10 accuracy in validation.Source: §4.1 · p. 6
  • The graph is updated independently of question answering, allowing pre-populated MKG retrieval to reduce QA latency and limit test-phase reliance on computation and search tools.When relevant information is absent from the MKG, additional search is still reported to cost less than using much larger models.Source: §4.1 · p. 6

4.2 Performance Comparison

  • 74.1% F1 on MEDQA and 66.34% accuracy on MedMCQA are reported for AMG-RAG, with both results surpassing larger models.The comparisons are reported in the MEDQA and MedMCQA benchmark tables.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • PubMed-MKG consistently outperforms Wiki-MKG, and ablating either CoT or MKG integration degrades accuracy and F1 on MEDQA.The confusion matrix compares AMG-RAG with and without CoT and knowledge-graph integration.Source: §4.2 · p. 7, Figure 3 · p. 7
  • On MEDQA, AMG-RAG with PubMed-MKG and an 8B backbone achieves 73.92% accuracy, while removing search or CoT lowers accuracy to 67.16% or 66.69%, respectively.The comparison also reports that PubMedSearch offers domain-specific, fresh evidence that general-purpose search modules may not provide.Source: Table 3 · p. 8, §4.2 · p. 8
  • AMG-RAG with GPT4o-mini and PubMed-MKG achieves 73.92% on MEDQA, and the paper reports superior performance in Neurology and Genetics domains.Switching to LLaMA 3.1 or Mixtral lowers performance under the same retrieval pipeline; Figure 4 compares question domains.Source: Table 4 · p. 8, §4.2 · p. 8, Figure 4 · p. 8

5 Conclusion

AMG-RAG dynamically constructs medical knowledge graphs and combines them with structured reasoning for medical QA. It reports improved accuracy and reasoning, outperforming similarly sized approaches and models 10 to 100 times larger.

  • AMG-RAG dynamically constructs medical knowledge graphs and integrates structured reasoning for medical question-answering.Source: §5 · p. 8
  • The system reports improved accuracy and reasoning capabilities, outperforming approaches of similar size and models 10 to 100 times larger.Source: §5 · p. 8

6 Limitations

AMG-RAG’s external search tools add latency when the graph is first built from scratch, and its applicability beyond medicine has not been explored. The approach also needs structured, authoritative medical sources, with structured access to treatment guidelines identified as future work.

  • External search tools add latency when the MKG is first built from scratch, but this occurs only once.Source: §6 · p. 9
  • AMG-RAG’s applicability to non-medical tasks remains unexplored.Source: §6 · p. 9
  • The approach needs structured, authoritative medical sources; future work should integrate structured access to treatment guidelines.The paper identifies guidelines as essential references for standardized diagnosis and treatment protocols.Source: §6 · p. 9

7 Ethics Statement

The ethics statement identifies inaccuracy and bias risks in medical QA and describes confidence scoring as a way to validate retrieved information. The graph assigns relationship scores using relevance criteria, while bias detection and mitigation remain active research areas.

7 Ethics Statement

  • Medical QA raises risks of inaccuracy and bias, and AMG-RAG uses MKG confidence scores to validate information from external sources.Source: §7 · p. 9
  • Bias detection and mitigation remain active research areas.Source: §7 · p. 9

A Confidence Scoring for the Relationships in the MKG

  • Relationship scores range from 1 to 10, with higher values indicating stronger relevance and lower values indicating weak or minimal connection.Scores of 10 denote direct, unambiguous relevance; 7–9 moderate to high relevance; 4–6 weak or tangential relevance; and 1–3 minimal or no meaningful connection.Source: A Confidence Scoring for the… · p. 11

B Evaluating the Accuracy and Robustness of the Medical Knowledge Graph

The MKG was assessed through expert-LLM ratings, blind testing, and qualitative review by medical experts. Reported findings include high component ratings, 89% accuracy in blind query testing, and expert feedback that refined graph relationships and confidence scores.

  • Expert LLMs rated node identification 8.9/10, relationship relevance 8.8/10, and node-summary clarity and precision 8.5/10.Source: B Evaluating the Accuracy and… · p. 11
  • 89% accuracy in blind tests of complex medical queries, with expert LLMs rating MKG interpretability and usability 9.4/10.Queries required multi-hop reasoning, including comorbidity management and multi-drug treatment protocols.Source: B Evaluating the Accuracy and… · p. 11
  • Medical experts assessed relationship correctness and completeness, multi-hop reasoning validity, and the graph’s utility for real-world diagnosis and treatment decisions.Source: B Evaluating the Accuracy and… · p. 11
  • Medical-expert feedback refined graph inconsistencies and confidence scores, while identifying contextual limitations that automated evaluations could miss.Experts noted contextual limitations in an example involving Diltiazem and Nitroglycerin.Source: B Evaluating the Accuracy and… · p. 11
  • Evaluations report that the MKG represented contraindications in specific patient populations and relationships across diverse medical domains, including cardiovascular treatments.Examples include beta-blockers Labetalol and Propranolol and treatments Diltiazem and Nitroglycerin.Source: B Evaluating the Accuracy and… · p. 12
  • The paper describes the MKG as human-readable and usable by advanced LLMs, with confidence scores and summaries supporting an interpretable representation of medical knowledge.Source: B Evaluating the Accuracy and… · p. 12

C QA Samples with reasoning from MEDQA benchmark

The MEDQA examples illustrate how AMG-RAG retrieves relevant information, organizes it into reasoning paths, and uses it to select an answer. One clinical case shows the model justifying its choice while addressing alternatives.

C QA Samples with reasoning from MEDQA benchmark

D Implementation Details for Dataset Ingestion and Vector Database

The implementation combines chunk-based vector retrieval with a medical knowledge graph. The graph contains many typed medical entities and relationships, while the ingestion and query workflow uses configured embedding, storage, and batching components.

D Implementation Details for Dataset Ingestion and Vector Database

D.1 Dataset Processing and Chunking

  • MEDQA textbook documents are divided into chunks of up to 512 tokens with a 100-token overlap to preserve context across boundaries.The overlap is described as supporting multi-hop reasoning over long documents.Source: §D.1 · p. 12

D.2 Embedding Model and Vector Storage

  • Text chunks and queries are embedded with SentenceTransformer’s all-mpnet-base-v2, and Chroma indexes the vectors while storing filenames and chunk IDs.The metadata supports document traceability.Source: §D.2 · p. 12

D.3 Batch Processing and Vector Database Population

  • Ingestion processes up to 10,000 chunks per batch and logs processed files to avoid redundant work, with error handling for failed processing.Source: §D.3 · p. 12

D.4 Query Answering Workflow

  • Queries are embedded with all-mpnet-base-v2, then Chroma similarity search returns the top k semantically relevant chunks for downstream QA.Source: §D.4 · p. 12

D.5 Key Configuration Details

  • The listed configuration specifies all-mpnet-base-v2, persistent Chroma storage, 512-token chunks with 100-token overlap, and batches of up to 10,000 chunks.Source: §D.5 · pp. 12–13

E.1 Neo4j

  • Neo4j is presented as a graph-based option for interconnected medical data, with horizontal scaling and cloud access described as supporting growth and distributed access.The section contrasts graph modeling with relational databases and notes that Neo4j stores entities as nodes linked by relationships and properties.Source: §E.1 · pp. 13–14
  • 76,681 nodes and 354,299 edges form a medical graph of typed relationships among entities including diseases, treatments, drugs, and clinical findings.The passage says this scale supports multi-hop reasoning and dynamic expansion and refinement.Source: §E.1 · p. 14

F Additional Results

The additional results section presents examples of AMG-RAG search items and structured reasoning for clinical questions. Captions identify the questions and their stated correct answers, but do not report comparative performance results.

F Additional Results

  • Table 7 presents search items and reasoning paths for an acute coronary syndrome question whose stated correct answer is Clopidogrel.Source: Table 7 · p. 15
  • Table 8 presents search items and reasoning paths for a copper IUD contraindication question, with active or recurrent pelvic inflammatory disease as the stated answer.Source: Table 8 · p. 16
  • Table 9 gives search items and reasoning paths for a postoperative confusion case whose stated answer is delirium.Source: Table 9 · p. 17
  • Table 10 gives search items and reasoning paths for a spermatogenesis question whose stated answer is primary spermatocyte.Source: Table 10 · p. 18
  • Table 11 is captioned as showing AMG-RAG answers with structured reasoning and citation-based grounding for clinical QA.Source: Table 11 · p. 18

Related papers

Loading 2502.13010v3…