Source-linked AI summary
ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge
Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, You Zhang
TL;DR
Medical-domain exploration by general-purpose LLMs is limited, while incorrect medical answers can harm patient treatment and well-being. The paper fine-tunes LLaMA on real patient-physician dialogues and adds autonomous retrieval from online and offline medical sources. ChatDoctor is reported to outperform ChatGPT on precision, recall, and F1 score and to answer questions about newer diseases and terms through retrieval.
Problem
General-purpose LLMs have limited medical-domain exploration, and wrong medical answers can be detrimental to patients’ treatment and well-being.
Method
The study fine-tunes LLaMA using real patient-physician conversations and equips ChatDoctor with autonomous retrieval from online and offline medical knowledge sources.
Results
ChatDoctor outperformed ChatGPT on precision, recall, and F1 score and answered questions about relatively new diseases and terms using retrieved information.
Takeaways & Limitations
ChatDoctor demonstrates a medical chatbot approach that combines patient-physician dialogue fine-tuning with retrieval of up-to-date medical knowledge.
Takeaways & Limitations
The model remains for academic research, and clinical use requires security measures such as automated reference checking and human expert evaluation.
Abstract
from arXiv · showhide
The primary aim of this research was to address the limitations observed in the medical knowledge of prevalent large language models (LLMs) such as ChatGPT, by creating a specialized language model with enhanced accuracy in medical advice. We achieved this by adapting and refining the large language model meta-AI (LLaMA) using a large dataset of 100,000 patient-doctor dialogues sourced from a widely used online medical consultation platform. These conversations were cleaned and anonymized to respect privacy concerns. In addition to the model refinement, we incorporated a self-directed information retrieval mechanism, allowing the model to access and utilize real-time information from online sources like Wikipedia and data from curated offline medical databases. The fine-tuning of the model with real-world patient-doctor interactions significantly improved the model's ability to understand patient needs and provide informed advice. By equipping the model with self-directed information retrieval from reliable online and offline sources, we observed substantial improvements in the accuracy of its responses. Our proposed ChatDoctor, represents a significant advancement in medical LLMs, demonstrating a significant improvement in understanding patient inquiries and providing accurate advice. Given the high stakes and low error tolerance in the medical field, such enhancements in providing accurate and reliable information are not only beneficial but essential.
Introduction
The study addresses limited medical-domain exploration by fine-tuning LLaMA-based dialogue models on real patient-physician conversations and adding online and offline knowledge retrieval. ChatDoctor combines these strategies to improve medical question answering and provide resources for further development.
- Medical-domain exploration of general-purpose LLMs remains relatively scarce despite their broad language capabilities.
- ChatDoctor was developed by fine-tuning LLaMA models on real-world patient-physician conversations and incorporating online or offline medical knowledge retrieval.
- 100K patient-physician conversations were used for training, while 10K conversations from a separate source were compiled for testing.
- The fine-tuned model outperformed ChatGPT on precision, recall, and F1 score according to the authors.
- The autonomous model retrieves online and offline medical knowledge to answer questions about up-to-date terms and diseases.
Materials And Methods
The method builds ChatDoctor from authentic patient-physician dialogues and an external medical knowledge brain. Prompts extract query keywords, retrieve relevant information, and support the model in producing answers within its token limit.
- Collection and preparation of patient-physician conversation dataset: Around 100K authentic patient-doctor interactions were collected from HealthCareMagic because synthetic dialogues could be overly specific and less diverse.
- Creation of external knowledge database: The external knowledge database covers diseases, symptoms, medical tests, treatment procedures, and medication suggestions.
- Creation of external knowledge database: ChatDoctor uses Wikipedia or a custom disease database as an external knowledge brain for retrieving information relevant to patient inquiries.
- Development of autonomous ChatDoctor with knowledge brain: Keyword-mining prompts first extract key terms from patient queries for knowledge search.
- Development of autonomous ChatDoctor with knowledge brain: A term-matching retrieval system returns top-ranked information from the knowledge brain based on the extracted keywords.
- Development of autonomous ChatDoctor with knowledge brain: Retrieved texts are divided to accommodate the LLM’s token-size limit before the model reads the domain knowledge and answers.
Results
ChatDoctor produced more medically specific or clinically congruent responses than ChatGPT across queries involving new terms, treatments, and patient symptoms. Its external knowledge retrieval supported accurate answers for Mpox, otitis, and Daybue, while several case responses more closely matched physician guidance.
- Medical knowledge retrieval: ChatDoctor retrieved information about Mpox and provided a precise answer, whereas ChatGPT could not recognize the term.The retrieved information came from Wikipedia through the model’s external knowledge feature.
- Medical knowledge retrieval: ChatDoctor gave a specialized response about otitis treatments, while ChatGPT provided a more general answer.The response followed retrieval of relevant medical knowledge.
- Medical knowledge retrieval: ChatDoctor accurately addressed Daybue’s purpose after autonomously retrieving relevant information, demonstrating an advantage over ChatGPT.Daybue had received FDA approval in March 2023.
- Patient cases: For unilateral headache, ChatDoctor proposed sinusitis as a possible cause, mirroring the physician’s diagnosis, while ChatGPT’s interpretation was incongruent.The case involved concern about a potential brain tumor.
- Patient cases: For a throat lump, ChatDoctor and iCliniq suggested enlarged lymph nodes and recommended biopsy and radiological diagnosis if initial treatments failed.ChatGPT only advised consultation with an ENT specialist.
- Patient cases: For exercise-related back pain, ChatDoctor generated an answer closer to iCliniq than ChatGPT.The pain intensified during breathing and torso or neck rotation, and the patient asked whether urgent attention was necessary.
- Patient cases: For blurred vision in a patient with retinal-detachment history, ChatDoctor and iCliniq identified retinal detachment or bleeding as potential issues, while ChatGPT did not speculate.All three sources advised professional ophthalmology consultation.
Discussion
ChatDoctor has potential uses in medical assessment and healthcare support, but the authors emphasize that it remains an academic research model requiring safeguards before clinical deployment. Human oversight and answer validation are needed because medical LLM errors can be harmful.
- Potential uses: ChatDoctor has potential uses ranging from preliminary patient assessment and automated case adjudication to proactive healthcare measures.The authors connect these uses to the model’s medical-data fine-tuning.
- Limitations: The model remains in the investigation phase and was developed for academic research only, not established clinical use.The authors specifically warn that clinical use is subject to wrong-answer risk.
- Limitations: Exclusive reliance on LLMs for medical diagnosis remains affected by false positives and false negatives.The authors state that concealed inaccuracies and hallucinations could lead to severe outcomes.
- Safeguards: Automated reference checking and human expert evaluation are needed to cross-validate answers, flag inaccuracies, and prevent hallucinations.The design and deployment of these security measures remains an open research topic.
Conclusions
With adequate training and online/offline supervision, ChatDoctor could improve diagnostic accuracy and efficiency while reducing medical professionals’ workload. The authors also identify possible benefits for access to medical consultations and future patient outcomes.
- With adequate training and online/offline supervision, ChatDoctor could potentially improve diagnostic accuracy and efficiency and reduce medical professionals’ workload.
- ChatDoctor may increase access to high-quality medical consultations, especially in underserved regions with limited medical resources.
- Further development and application of ChatDoctor may eventually improve patient outcomes and advance medical research.
Additional Information
The authors report that the study involved neither human nor animal participants or tissue. They also disclose NIH support and refer to the ICMJE uniform disclosure form for conflicts of interest.
- The study did not involve human participants or tissue.
- The study did not involve animal subjects or tissue.
- The work was supported by National Institutes of Health grants R01 CA240808 and R01 CA258987.
- The authors reported financial relationships using the ICMJE uniform disclosure form.