Source-linked AI summary
DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task
Honglin Xiong, Sheng Wang, Yitao Zhu, Zihao Zhao, Yuxiao Liu, Linlin Huang, Qian Wang, Dinggang Shen
TL;DR
General LLMs are less precise in medical contexts and typically perform better in English, while hospitals are thought to face prohibitive training and deployment costs. DoctorGLM addresses these issues by collecting Chinese medical dialogues and fine-tuning ChatGLM-6B with efficient tuning and quantization techniques. The paper reports an affordable healthcare-focused LLM pipeline but describes the system as an early-stage engineering attempt with substantial limitations.
Problem
General LLMs lack explicit medical training, typically perform better in English, and may provide suboptimal medical advice, while healthcare organizations face substantial customization costs.
Method
DoctorGLM collects Chinese medical dialogues and fine-tunes bilingual ChatGLM-6B using parameter-efficient tuning, quantization, prompt design, and ChatGPT-assisted data construction.
Results
The paper reports that a specialized Chinese medical dialogue model can be trained for less than 100 USD.
Takeaways & Limitations
The authors present DoctorGLM as an affordable, customizable healthcare LLM pipeline applicable across languages and clinical departments.
Takeaways & Limitations
DoctorGLM is an early-stage engineering attempt containing numerous mistakes and is unsuitable for commercial or clinical use.
Abstract
from arXiv · showhide
The recent progress of large language models (LLMs), including ChatGPT and GPT-4, in comprehending and responding to human instructions has been remarkable. Nevertheless, these models typically perform better in English and have not been explicitly trained for the medical domain, resulting in suboptimal precision in diagnoses, drug recommendations, and other medical advice. Additionally, training and deploying a dialogue model is still believed to be impossible for hospitals, hindering the promotion of LLMs. To tackle these challenges, we have collected databases of medical dialogues in Chinese with ChatGPT's help and adopted several techniques to train an easy-deploy LLM. Remarkably, we were able to fine-tune the ChatGLM-6B on a single A100 80G in 13 hours, which means having a healthcare-purpose LLM can be very affordable. DoctorGLM is currently an early-stage engineering attempt and contain various mistakes. We are sharing it with the broader community to invite feedback and suggestions to improve its healthcare-focused capabilities: https://github.com/xionghonglin/DoctorGLM.
1. Introduction
DoctorGLM addresses the limited medical and Chinese-language suitability of general LLMs by fine-tuning a bilingual medical model and developing an affordable customization pipeline. The paper presents this approach as making healthcare LLM training and deployment accessible to hospitals.
- General LLMs are not specifically designed for medicine and typically perform better in English, limiting the precision and accessibility of medical advice.
- DoctorGLM fine-tunes the bilingual ChatGLM-6B model to optimize medical dialogue in Chinese and English.ChatGLM-6B is a pretrained model with 6 billion parameters.
- Parameter-efficient tuning and quantization techniques enable a specialized Chinese medical dialogue model to be trained for less than 100 USD.
- The paper presents a comprehensive dialogue-model training pipeline applicable across languages and adaptable to specific clinical departments.The source code is made available on GitHub.
- The authors demonstrate that training and deploying a personalized LLM can be affordable enough for hospitals to use inhouse data.
2. Large Language Models in Healthcare
Healthcare language models have progressed from biomedical-domain encoders to larger conversational systems, while translation and dataset construction support specialized non-English applications. The section situates DoctorGLM within this development and describes the datasets used.
- BioBERT, PubMedBERT, and ClinicalBERT adapt BERT to biomedical or clinical corpora, with ClinicalBERT outperforming its predecessor.
- Table 1 reports dataset statistics and identifies multi-round QA, language-model-generated data, and Chinese medical dialogue.
- Med-PaLM achieved 67.6% accuracy on the MedQA exam, while ChatGPT exceeded 50% accuracy across USMLE exams without supplementary medical training.
- ChatCAD combines medical image analysis models with ChatGPT to provide interactive computer-aided diagnosis, while ChatDoctor fine-tunes LLaMA using ChatGPT-synthesized clinical QA.
3. Approach
DoctorGLM builds medical dialogue data with ChatGPT, translates English resources through a low-cost distillation pipeline, and improves responses with disease-specific prompts. It then fine-tunes ChatGLM-6B using parameter-efficient adaptation.
- Dataset construction: The translation pipeline selects raw English text X from ChatDoctor and obtains corresponding high-quality translations Y through the ChatGPT API.
- Dataset construction: A BART-based model is fine-tuned on ChatGPT translations of paired English source and target texts, distilling ChatGPT’s knowledge into a smaller model.
- Dataset construction: ChatGPT-generated instruction data uses disease, symptom, and drug information to preserve conversational fluency while improving diagnostic accuracy over C
- Prompt designer: DoctorGLM uses a prompt designer that extracts disease or symptom keywords and adds disease-library information about symptoms, diagnosis, treatment, and prevention.The library contains 3231 disease documents sourced from the Merck Manual of Diagnosis and Thera.
- Prompt designer: The generated disease prompt is combined with the original user input to improve response accuracy and reliability, and the approach can also be used with other LLMs.
- Model and fine-tuning: ChatGLM-6B is an open bilingual model with 6.2 billion parameters, optimized for Chinese question answering and dialogue.
- Model and fine-tuning: Fine-tuning applies low-rank adaptation to q and v with rank 8 and alpha 16 on an A100 GPU.The reported training duration is 8 hours.
4. Main Results
DoctorGLM is compared with general-purpose models and evaluated with parameter-efficient fine-tuning methods. The reported results highlight differences in response conservativeness and a training failure after prolonged iterations.
- DoctorGLM tends to be more arbitrary in its medical responses, whereas ChatGPT is more conservative.
- 4.2. Generation Diversity: Top-p and temperature are used to control the diversity and quality of generated responses.Top-p selects a probability mass threshold, while temperature adjusts output randomness.
- 4.3. How many iteration is good enough?: DoctorGLM experienced model collapse after 50K iterations and could not produce meaningful output.
- 4.4. P-tuning vs. LoRA: P-tuning optimizes continuous prompts while using only 0.1%-3% of the fine-tuning parameters.The paper compares P-tuning V2 with LoRA for DoctorGLM.
5. Cost
Fine-tuning DoctorGLM is presented as substantially cheaper than training an LLM from scratch. The reported costs cover both training on an A100 GPU and inference on a consumer-level GPU.
- 80,000 question-and-answer pairs can be processed per hour per GPU, making fine-tuning 100,000 pairs cost approximately 18.75 USD.This estimate assumes three epochs and an A100 cloud cost of approximately 5 USD per hour.
- DoctorGLM inference requires about 13 GB of GPU memory and can run on an RTX 3090.The estimated inference PC cost is approximately 1500 USD, with costs varying by GPU prices and location.
- Table 2 marks doctors’ comments in blue, factual errors in red, and improper diagnoses in green.
6. Technical Limitations
The paper identifies several technical limitations affecting DoctorGLM’s reliability, speed, deployment, and training stability. These issues constrain its current practical use and require further engineering work.
- DoctorGLM is an early-stage work containing numerous mistakes and is unsuitable for commercial or clinical use.
- DoctorGLM can lose capability during fine-tuning and occasionally repeat itself.The authors suspect fine-tuning may incur a higher alignment cost than RLHF.
- Responses take approximately 15 to 50 seconds, depending on token length, and are slower than ChatGPT through its web API.The delay is partly attributed to the chatbot’s typing indicator.
- Quantization difficulties prevent deployment of the trained DoctorGLM on more affordable GPUs such as the RTX 3060 and RTX 2080.ChatGLM runs satisfactorily on INT-4 using about 6G, but the trained LoRA has issues.
- Model performance declines with prolonged training, but the authors lack a strategy for deciding when to stop.They state that cross-entropy appears to be an overly rigid constraint when fine-tuning LLMs.