MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs

• Medical CoT data are scarce and difficult to scale, while existing generation approaches can omit quality filtering or produce factual errors that undermine clinical applicability.The introduction contrasts outcome-filtered BOLT with limited medical-specific reasoning and gives an example of GPT-4o misstating steroid treatment for ARDS.Source: §1 · p. 2
• The pipeline maps question and answer entities to medical knowledge-graph nodes, finds and prunes connecting paths, and uses the remaining paths to scaffold medically grounded explanations.It draws QA pairs from seven datasets and uses exact matching or LLM-based similarity selection to map entities.Source: §1 · pp. 2–3
• Knowledge-graph grounding is intended to keep reasoning steps logically coherent and clinically valid, and MedReason outperforms existing datasets in automated metrics and physician evaluations across seven specialties.Side-by-side comparisons also indicate more factually precise and clinically supportive reasoning chains than competing approaches.Source: §1 · pp. 2–3
• A verification step has an LLM answer each question using the generated path, removing samples that fail to yield correct answers and retaining 32,682 CoT samples.The retained paths are described as logically sound and clinically valid.Source: §1 · p. 3
• MedReason achieves state-of-the-art performance among 7–8B models on challenging clinical benchmarks, while SFT with its data improves diverse base models across benchmarks.The evaluation covers seven QA benchmarks, including MedBullets, MedXpert, and Humanity’s Last Exam.Source: §1 · p. 3

Reasoning with knowledge in LLMs

• Medical LLMs need high-quality intermediate reasoning data, but manual annotation does not scale and model distillation can produce hallucinations and inaccurate medical reasoning.Prior reasoning LLM progress in math and coding motivated analogous development for medicine (Guo et al., 2025; Team et al., 2025; Jaech et al., 2024; Li et al., 2025; Goh et al., 2024; Lucas et al., 2024).Source: §2 · p. 3

LLM-Distilled Medical Datasets

• Earlier medical LLM datasets focused on instruction tuning, while Chen et al. (2024b) used GPT-4o strategy-based retrieval to create 20K question-answer pairs with complex chain-of-thought data.General-domain LLMs can fail to preserve medical knowledge at every reasoning step.Source: §2 · pp. 3–4
• MedReason uses a medical knowledge graph to provide factual guidance during medical chain-of-thought generation, building on work using dependable information for multigranular medical-image captions (Xie et al., 2024c).Source: §2 · p. 4

3.1.1 Medical Entity Extraction and Mapping

• Using GPT-4o (Hurst et al., 2024) and PrimeKG, the pipeline extracts Q/A medical entities and maps them by exact match, similarity threshold, then LLM fallback.The similarity threshold is 0.85.Source: §3.1 · p. 4, §3.1.1 · pp. 4–5

3.1.2 Paths Searching and Pruning

• For each mapped question–answer entity pair, they find shortest KG paths and use an LLM to select the three most question-relevant paths as concise evidence for CoT.The selected paths are aggregated across entity pairs for CoT generation.Source: §3.1.2 · p. 5

3.2 CoT Generation and Quality Filtering

• LLMs elaborate relevant KG paths into clinically grounded CoT, then retain only chains whose internally generated answers match ground truth: 32K of 45K samples.The generated reasoning steps align with KG-derived evidence, and answer agreement is used to filter for logical validity and factual accuracy.Source: §3.2 · pp. 5–6

4.1 Experimental Setup

• The pipeline used training-set QA pairs, reducing 55K gathered pairs to 45K eligible for CoT generation after excluding pairs without entities in the question or answer.The source datasets included MedQA, MedMCQA, PubmedQA, MMLU, HuatuoGPT-o1, MedXpert, and Humanity’s Last Exam.Source: §4.1 · p. 6
• Evaluation covered standard medical QA benchmarks and challenging clinical, expert-level reasoning, and recent medical-question benchmarks.The standard benchmarks were MedQA, MedMCQA, health and biology tracks of MMLU-Pro, and PubMedQA; additional evaluations included MedBullets, MedXpert, and HLE. The study fine-tuned instruction-tuned LLaMA 3.1-Instruct-8B and Mistral-Instruct-7B, as well as Medical-CoT-8B, DeepSeek-Distill-8B, and Huatuo-o1-RL-8B.Source: §4.1 · p. 6

4.2 Results

• 57.3% average score placed MedReason-8B ahead of all other models across five evaluation datasets, 1.4% above its Huatuo-o1-RL-8B base model.It also outperformed Huatuo-o1-RL-8B by 4.2% on MedBullets and 2.3% on MedXpert.Source: §4.2 · pp. 7–8, Table 4 · p. 8
• MedReason fine-tuning raised average accuracy from 45.8% to 51.2% (+5.4%) for Llama3.1-Instruct-8B and from 36.1% to 44.7% (+8.6%) for Mistral-Instruct-7B.The gains exceeded those from Huatuo CoT data, which were +4.1% and +6.0%, respectively.Source: §4.2 · pp. 6–7, Table 2 · p. 7
• On challenging clinical datasets, MedReason fine-tuning improved average scores by 5.3% for Medical-CoT-8B and 7.7% for DeepSeek-Distill-8B.On general medical QA datasets, the average gains were 5.3% and 3.5%, respectively.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 Ablation Study and Expert Verification

• Quality filtering increased the LLaMA3.1-Instruct-8B average score by 1.1% and improved performance across most medical datasets.The ablation evaluated whether filtering generated CoT data for quality affected performance.Source: §4.3 · p. 8, Table 5 · p. 8
• In a MedBullets case study, MedReason correctly diagnosed Cri-du-Chat syndrome from a chromosome 5p deletion, while DeepSeek-distilled expressed uncertainty and HuatuoGPT-o1 gave an incorrect diagnosis.The diagnosis matched the ground-truth answer.Source: §4.3 · p. 8, Figure 4 · p. 9
• Clinicians across seven specialties preferred MedReason’s CoT explanations, with Gastroenterology experts unanimously selecting it (100%) and Dermatology and Oncology experts showing over 80% preference.Experts compared anonymized MedReason and HuatuoGPT-o1 explanations for more than 25 randomly sampled questions per specialty.Source: §4.3 · pp. 8–9, Figure 5 · p. 9

5 Conclusion

• The pipeline uses structured knowledge graphs to generate medically grounded, interpretable Chain-of-Thought data intended to enhance clinical validity.It is designed to scale automatic generation of high-quality medical reasoning data.Source: §5 · p. 9
• Experiments with instruction-tuned and reasoning-specialized LLMs show consistent improvements across medical benchmarks, particularly in complex clinical scenarios.The passage does not specify benchmark names or numerical gains.Source: §5 · p. 9
• Expert evaluations find the generated reasoning quality superior to prior methods, supporting further exploration of clinically valid reasoning for trustworthy medical AI.Source: §5 · p. 9

A.1 OpenAI API Usage

• The pipeline used Azure GPT-4o version ’gpt-4o-0806-nofilter-global’ with API version ’2024-12-01-preview’, and total API usage was about $3,600.Source: §A.1 · p. 14

A.2 Data Statistics

• Table 6 summarizes raw, generated, and final quality-filtered QA data; HLE and MedXpert held out the remaining samples for testing, while other sources used training data only.The training allocations were 57 samples for HLE and 666 for MedXpert.Source: Table 6 · p. 14, §A.2 · p. 14

A.3 Prompts

• The pipeline prompts entity identification in both the question and answer.Figure 6 shows the entity-identification prompt.Source: §A.3 · p. 14, Figure 6 · p. 14
• A separate prompt selects the most relevant knowledge-graph nodes for the extracted entities.Figure 7 shows the node-selection prompt.Source: §A.3 · p. 14, Figure 7 · p. 15
• The pipeline prompts pruning of irrelevant paths before generating reasoning data.Figure 8 shows the path-pruning prompt.Source: §A.3 · p. 14, Figure 8 · p. 15
• It also prompts chain-of-thought generation with paths and answer generation from that chain-of-thought data for quality filtering.Source: §A.3 · p. 14

A.4 Detailed Algorithm

• Text embeddings rank graph-node candidates for each extracted entity, and the algorithm forms a Top-K candidate set for mapping.The algorithm takes the extracted entity set, knowledge graph, embedding model, similarity threshold, and LLM selection function as inputs.Source: §A.4 · pp. 15, 17
• Selection proceeds by case-insensitive exact match, then top-candidate similarity above τ = 0.85, and finally LLM judgment using question-answer context when earlier stages fail.The LLM chooses the most relevant candidate node from the set if neither exact nor threshold-based matching selects one.Source: §A.4 · pp. 16–17

A.5 Detailed comparison between MedReason-8B and Huatuo-o1-RL-8B.

• MedReason-8B correctly linked vomiting with hyperglycemia, while Huatuo-o1-RL-8B overlooked the link and answered incorrectly.The comparison is shown in Figure 11.Source: §A.5 · p. 17, Figure 11 · p. 18