Source-linked AI summary
DocTalk: Enhancing Medical Knowledge Retrieval via Fine-tuned Large Language Models with Chain-of-Thought Reasoning for Robust Clinical Decision Support
Asad Gohar, Md Ariful Islam Mozumder, Moon-Il Joo, Hee-Cheol Kim
TL;DR
Medical LLMs need demonstrated accuracy, impartiality, and patient safety before real-world use. DocTalk generates chart-grounded QA from discharge summaries and instruction-tunes open models; performance improved on DocTalk-QA and transferred to PubMedQA, with gains varying by model.
Problem
Before medical LLMs are used in real situations, their accuracy, impartiality, and patient safety must be checked.
Source: §1 · p. 2Method
DocTalk generates high-fidelity QA pairs from de-identified discharge summaries and fine-tunes open-source LLMs with a lightweight LoRA-based framework.
Source: §6 · p. 15Results
Instruction tuning improved DocTalk-QA metrics and transferred to PubMedQA, where Llama-3.1-8B-Instruct accuracy rose from 0.931 to 0.977.
Source: §4.2 · p. 11, §5 · p. 15Takeaways & Limitations
DocTalk was deployed as a secure web platform for interactive clinician engagement with electronic health records.
Source: §5 · p. 14, §6 · p. 16Training and evaluation QA pairs came mainly from one healthcare organization, and the system focuses on single-turn question answering.
Source: §5 · p. 15
Abstract
from arXiv · showhide
Objective: The rapid advancement of large language models (LLMs) in healthcare has opened new possibilities for AI-assisted clinical decision-making; however, safe and effective deployment remains challenging because of hallucination and domain misalignment. This study introduces DocTalk, a doctor-assistive question-answering (QA) system designed to generate concise, chart-grounded responses from de-identified discharge summaries. Methods: We developed a high-fidelity data generation pipeline that produces 500 clinically relevant QA pairs through sentence-level decomposition, chain-of-thought-guided question formulation, and expert review. Six open-source LLMs, including three general-purpose and three biomedical models, were evaluated before and after parameter-efficient instruction tuning using Low-Rank Adaptation (LoRA). The framework integrates prompt engineering, in-context learning, and medically aligned evaluation to support factual and interpretable responses. Results: On DocTalk-QA, instruction tuning produced consistent improvements across lexical, semantic, and clinical fidelity metrics. The best-performing tuned model, Llama-3-Med42-8B, achieved ROUGE-L = 0.40, BERTScore-F1 = 0.76, and UMLS-F1 = 0.49. To examine external generalization beyond the DocTalk-QA dataset, the two best-performing models, Llama-3.1-8B-Instruct and Llama-3-Med42-8B, were evaluated on PubMedQA before and after instruction tuning. Accuracy improved from 0.931 to 0.977 for Llama-3.1-8B-Instruct and from 0.935 to 0.943 for Llama-3-Med42-8B. Conclusion: DocTalk was deployed as a secure web-based platform for interactive EHR querying. These findings suggest that clinically grounded instruction tuning can improve medical QA performance and support practical clinical decision support.
1. Introduction
Clinical QA must retrieve useful details from complex EHR narratives while meeting accuracy and safety expectations. DocTalk builds clinician-oriented, chart-grounded QA from de-identified clinical notes and adapts open-source models for this task.
1. Introduction
- EHRs contain extensive unstructured medical information, making efficient extraction difficult, while complex language can also be hard for patients to understand.LLMs are proposed as a way to retrieve structured clinical information from unstructured reports.Source: §1 · p. 1
- Healthcare LLMs require rigorous evaluation for accuracy, fairness, and patient safety across established medical benchmarks and topics.Existing medical QA benchmarks include MedQA [23], emrQA [25], MedMCQA, PubMedQA, and MMLU [28].Source: §1 · p. 2
- Patient-facing summaries may omit details clinicians need, while exam-style QA often fails to mimic EHR settings and can cover only a few fields.Clinical models may also struggle with out-of-distribution notes across institutions or templates [30].Source: §1 · p. 2
- DocTalk splits clinical notes into atomic sentences, formulates doctor-style questions with a brief chain-of-thought prompt, and adapts six LLMs using LoRA.The six models include three general-purpose and three biomedical models.Source: §1 · pp. 2–3
2. Related Works
Related work spans healthcare LLMs, EHR question answering, instruction tuning, efficient adaptation, and biomedical QA benchmarks. Across these areas, DocTalk targets clinician-verified questions from de-identified EHR discharge summaries and concise, chart-grounded answers.
2.1. LLMs in Healthcare
- Healthcare LLMs have shown promise, but safety, interpretability, and heterogeneous EHR documentation remain challenges for real-world deployment.Earlier models showed strong standardized-benchmark performance [41], while later analyses identified memorization, factual inconsistency, and distributional-shift concerns [42].Source: §2.1 · p. 3
2.2. Clinical Question Answering over EHR Narratives
- EHR QA requires document-level retrieval and nuanced clinical detail; patient-facing discharge summaries may not meet clinicians’ decision-support needs.emrQA created question-answer pairs from medical records [44], while earlier LSTM approaches struggled with long-context reasoning and document coherence.Source: §2.2 · p. 3
2.3. Instruction-Tuned Clinical LLM
- Instruction tuning can improve medical reasoning, but benchmark performance does not ensure generalization to real-world EHR narratives.Targeted medical fine-tuning improved reasoning and diagnostic accuracy, while later healthcare-specialized models still often relied on exam-style data [48].Source: §2.3 · p. 4
2.4. Prompting Strategies and Parameter-Efficient Adaptation
- Chain-of-thought prompting supports stepwise reasoning, while LoRA specializes models by updating a small parameter subset to reduce computational overhead.Self-consistency and multi-sample reasoning aggregate multiple reasoning paths [50].Source: §2.4 · p. 4
2.5. LLMs in Biomedical Question Answering
- Biomedical QA research includes domain-adapted foundation models, but DocTalk distinguishes itself by using clinician-verified EHR questions and concise, chart-grounded answers.BioMistral showed improved biomedical-language performance after targeted medical-corpus pretraining [61].Source: §2.5 · p. 4
3. Materials and Methods
DocTalk combines de-identified discharge/EHR notes, prompting, LoRA instruction tuning, evaluation on clinical and biomedical QA tasks, and secure platform deployment.
3. Materials and Methods
- The study evaluates a five-component workflow spanning DocTalk-QA construction, prompting, LoRA tuning, internal and external evaluation, and secure deployment.The framework targets concise, chart-grounded answers.Source: §3 · p. 4
3.1. Datasets and Tasks
- MIMIC-IV supplies de-identified EHR data, while PubMedQA tests generalization to biomedical research questions using abstracts as context.The datasets support comparison across different document types and reasoning settings.Source: §3.1 · p. 5
3.2. Data Pre-processing and Tokenization
- Discharge notes were standardized, stripped of artifacts, medically normalized, tokenized, and frequency-filtered to retain clinical terms.A biomedical-specific tokenizer was used, and special tokens supported unknown words and sequence boundaries.Source: §3.2 · p. 6
3.3. DocTalk-QA Dataset from MIMIC-IV
- MIMIC-IV notes were segmented into coherent sentences, then Llama-3.3-70B-Instruct generated clinician-style questions using CoT prompting.Outputs were string-matched against source text; the model was selected for clinical phrasing adherence and 128K-token context capacity.Source: §3.3 · p. 6
3.4. Dataset Overview
- The resulting 500 QA pairs underwent licensed-clinician review for correctness, completeness, safety, and clinician-needs alignment.ICD-10 checks and UMLS validation supported diagnostic accuracy and biomedical concept grounding.Source: §3.3 · p. 6
3.5. Pretrained LLMs and Optimization
- Six general-purpose and biomedical LLMs were adapted with LoRA; across models, approximately 0.36% of parameters were updated.LoRA tuning used rank 16, alpha 32, and dropout 0.05.Source: §3.5 · pp. 8–9
3.6. Few-Shot and Chain-of-Thought Strategic Prompting
- CoT generated reasoning-aware QA pairs, while training used two training-split exemplars per item across diverse clinical question types.The exemplars included discharge-summary excerpts, clinician-style questions, and short chart-grounded answers.Source: §3.6 · p. 9
3.7. Evaluation Settings
- Training masked loss to answer tokens, while evaluation measured lexical overlap, semantic similarity, and UMLS concept-level clinical correctness.The metrics included ROUGE-1, ROUGE-2, ROUGE-L, BERTScore-F1, and UMLS-F1.Source: §3.7 · p. 10
4. Experimental Results
Instruction tuning improved DocTalk-QA performance, especially clinical fidelity, and both selected models gained on external PubMedQA, though the gains differed substantially.
4.1. Pretrained Model Evaluation on DocTalk-QA
- Before tuning, Llama-3.1-8B-Instruct led DocTalk-QA baselines with ROUGE-L 0.24, BERTScore-F1 0.69, and UMLS-F1 0.23, while baseline UMLS-F1 ranged from 0.11 to 0.23.Source: §4.1 · p. 11, Table 2 · p. 11
4.2. Instruction-Tuned Model Performance on DocTalk-QA
- Across six models, tuning raised average ROUGE-L by 0.18, BERTScore by 0.09, and UMLS-F1 by 0.26; Llama-3-Med42-8B reached ROUGE-L 0.40, BERTScore-F1 0.76, and UMLS-F1 0.49.The average gain was larger for clinical fidelity than for lexical or semantic metrics.Source: §4.2 · p. 11, Table 3 · p. 12
4.3. Performance Evaluation on PubMedQA
- Accuracy rose from 0.931 to 0.977 for Llama-3.1-8B-Instruct and from 0.935 to 0.943 for Llama-3-Med42-8B on PubMedQA after tuning.PubMedQA evaluation used binary Yes/No classification despite a 92.8% positive-class imbalance; the authors selected MCC as the primary discrimination metric, and Llama-3-Med42-8B was not necessarily strongest on this external benchmark.Source: §4.3 · pp. 12–13, Table 4 · p. 13
5. Discussion
The discussion presents DocTalk as a clinically grounded, chart-focused platform and reports cross-benchmark gains whose magnitude varies by model. It also identifies data, reasoning, and language boundaries that constrain its current scope.
5. Discussion
- DocTalk combines authentic clinical narratives, atomic-sentence QA construction with manual quality assurance, and system grounding, few-shot examples, answer-only supervision, and LoRA tuning.The authors contrast this data-centric approach with prior work using simulated datasets or benchmark QA pairs detached from practical clinical use [22] [73] [74].Source: §5 · p. 14
- The web platform supports secure login, patient-report selection, and natural-language EHR queries; concise, interpretable, chart-faithful answers target real-time clinical use.The authors describe this approach as feasible with limited resources and suitable for point-of-care integration.Source: §5 · pp. 14–15
- Accuracy rose from 0.931 to 0.977 for Llama-3.1-8B-Instruct on PubMedQA; Llama-3-Med42-8B improved less, showing model-dependent transfer.The best-performing model on in-domain DocTalk-QA was not necessarily the best on external PubMedQA.Source: §5 · p. 15
- Training and evaluation rely mainly on discharge summaries from one healthcare organization, limiting demonstrated generalizability across hospitals, specialties, and documentation styles.The authors also report errors in complex comorbidity or reasoning scenarios, single-turn-only QA, and English-only development and evaluation.Source: §5 · p. 15
- Future work includes patient-data summaries, diagnostic recommendations, temporal-report tracking, clinician feedback, and large-scale prospective clinical trials.The authors identify professional feedback and prospective trials as needed for safe, effective real-world deployment.Source: §5 · p. 15
6. Conclusion
DocTalk uses a lightweight LoRA-based framework to generate clinically grounded QA from de-identified discharge summaries. The authors report gains on clinical and textual metrics, as well as external PubMedQA improvement that varies across model architectures.
6. Conclusion
- Llama-3-Med42-8B and other DocTalk models outperform others on both text-based and clinically aligned metrics.The approach generates high-fidelity QA pairs from de-identified discharge summaries and fine-tunes open-source LLMs with lightweight LoRA.Source: §6 · p. 15
- DocTalk-based instruction tuning improved biomedical reasoning beyond the original clinical QA setting, with transfer magnitude varying across model architectures.This finding is based on assessment with the external PubMedQA benchmark.Source: §6 · p. 16
- The deployed web platform supports interactive clinician engagement, and the authors describe the system as a scalable solution for healthcare applications.The platform is presented alongside contributions in data generation, instruction tuning, and evaluation.Source: §6 · p. 16