Source-linked AI summary
ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation
Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan, Xiaohu Li
TL;DR
General large language models remain limited in medical applications because of factual, reasoning, and real-world-grounding challenges. ClinicalGPT addresses this gap by fine-tuning on diverse medical data and evaluating clinical capabilities across multiple tasks, with results showing higher performance than comparison models but weaker results in some disease categories.
Problem
General large language models have limited effectiveness in medical applications because of factual inaccuracies, reasoning challenges, and insufficient grounding in real-world experience.
Method
ClinicalGPT is fine-tuned with diverse medical records, knowledge-graph data, examinations, and multi-turn medical dialogues, then evaluated across several clinical tasks.
Results
ClinicalGPT outperformed comparison models across medical examinations, diagnosis, question answering, and medical conversation evaluations.
Takeaways & Limitations
The results support adapting large language models with diverse clinical data and comprehensive task-specific evaluation for medical applications.
Takeaways & Limitations
ClinicalGPT performed less well in Respiratory and Digestive diseases, indicating a need for further refinement in those categories.
Abstract
from arXiv · showhide
Large language models have exhibited exceptional performance on various Natural Language Processing (NLP) tasks, leveraging techniques such as the pre-training, and instruction fine-tuning. Despite these advances, their effectiveness in medical applications is limited, due to challenges such as factual inaccuracies, reasoning abilities, and lack grounding in real-world experience. In this study, we present ClinicalGPT, a language model explicitly designed and optimized for clinical scenarios. By incorporating extensive and diverse real-world data, such as medical records, domain-specific knowledge, and multi-round dialogue consultations in the training process, ClinicalGPT is better prepared to handle multiple clinical task. Furthermore, we introduce a comprehensive evaluation framework that includes medical knowledge question-answering, medical exams, patient consultations, and diagnostic analysis of medical records. Our results demonstrate that ClinicalGPT significantly outperforms other models in these tasks, highlighting the effectiveness of our approach in adapting large language models to the critical domain of healthcare.
1 Introduction
ClinicalGPT addresses the limited effectiveness of general large language models in healthcare by adapting them specifically to medical applications. It combines diverse medical data with broad clinical evaluation, and the reported results show higher performance than existing models.
- Generic medical language-model performance remains limited by factual inaccuracies, reasoning challenges, and insufficient grounding in real-world experience.
- ClinicalGPT is designed for tasks across medical applications, using extensive real-world medical records to transfer domain-specific knowledge into the model.
- The model is evaluated across medical knowledge question-answering, examinations, patient consultations, and medical record analysis.
- ClinicalGPT outperforms existing models in the reported medical tasks, supporting the effectiveness of its clinical adaptation.
2 Methods
ClinicalGPT is trained through text generation and instruction tuning on diverse medical datasets, including question-answer pairs, examinations, dialogues, and electronic health records. Its pipeline further incorporates knowledge-graph supervision, reward modeling, and reinforcement learning with a KL-divergence constraint.
- 2.1 Dataset: ClinicalGPT combines medical question-answering, examination, dialogue, and electronic health-record datasets for training and evaluation.The datasets include cMedQA2, cMedQA-KG, MEDQA-MCMLE, MedDialog, and MD-EHR.
- 2.2 Finetuning: The model uses supervised instruction tuning to maximize the likelihood of generating a response conditioned on an input prompt.The response and prompt are represented as token sequences, with lengths n and m respectively.
- 2.2 Finetuning: Knowledge-graph triples are converted with manually designed templates into question-answer pairs for instruction tuning.The triples represent entities and relations such as disease, medication, and symptom relationships.
- 2.3 Reward model: Human labelers rank response pairs to train a reward model that supplies feedback for reinforcement learning.The reward-model loss favors the human-preferred response over the less-preferred response.
- 2.4 Reinforcement learning: Proximal Policy Optimization penalizes KL divergence from the supervised model to limit excessive policy deviation during reinforcement learning.The reward combines the learned reward-model score with a KL-divergence penalty.
3 Experiments and results
ClinicalGPT was evaluated across medical conversation, examination, electronic-record diagnosis, and medical question-answering tasks. It generally outperformed comparison models, while showing narrower weaknesses in respiratory and digestive examination categories and in some specialty diagnosis groups.
- Medical conversation: ClinicalGPT achieved outstanding BLEU-1 and ROUGE performance in medical conversation, ranking second to BLOOM-7B on BLEU-2, BLEU-3, and BLEU-4.The evaluation used truncated multi-round MedDialog histories and BLEU, ROUGE, and GLEU metrics.
- Medical examination: 38.4 average examination accuracy exceeded ChatGLM-6B by 19.9, BLOOM-7B by 25.7, and LLAMA-7B by 27.2.The strongest category was Rheumatic immune at 47.4, while Respiratory and Digestive reached 26.1 and 36.9.
- Diagnosis: 80.9% average diagnostic accuracy exceeded ChatGLM-6B at 40.9%, LLAMA-7B at 36.6%, and BLOOM-7B at 60.3% across disease groups.ClinicalGPT reached 90.1% in Digestive and 89.9% in Urinary diagnosis, with lower results in Gynecology and Hematology.
- Medical question answering: 89.7% and 85.0% of medical question-answering comparisons were wins against BLOOM-7B and LLAMA-7B, while wins against ChatGLM-6B reached 67.2%.Tie rates were 1.8% and 2.3% against BLOOM-7B and LLAMA-7B, and 10.9% against ChatGLM-6B.
4 Conclusion
ClinicalGPT is tailored to medical and clinical applications through diverse fine-tuning data and broad task evaluation. Experiments report superior capabilities in medical conversation, examination, diagnosis, and question answering.
- ClinicalGPT’s experiments report superior capabilities across medical conversation, examination, diagnosis, and medical question answering.
A Medical knowledge graphs
The paper uses three medical knowledge graphs with varied entity and relationship coverage. Together, they provide structured information spanning diseases, symptoms, examinations, medications, and treatments.
- CMeKG contains 62k entities and 374k relationship triplets across nine medical entity types and 23 relationship types.
- xywy-KG contains 44k entities and 294k relationships derived from a Chinese online medical consultation website.
- 39Health-KG comprises 37k entities and 210k entity relationships spanning seven entity types and eight relationship types.
B Prompt templates
The method transforms structured medical knowledge into question-answer training data using prompt templates. Table 8 provides examples of these templates.
- Prompt templates transform knowledge triplets into question-answer data for training language models.
- Table 8 presents examples of the prompt templates used for this transformation.