Source-linked AI summary
HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge
Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, Ting Liu
TL;DR
Biomedical tasks require specialized expertise that general-domain LLMs may lack, especially in Chinese contexts. HuaTuo addresses this gap by fine-tuning LLaMA-7B with CMeKG-based instruction data, and the reported evaluation shows improved medical knowledge usability without much compromising safety.
Problem
General-domain LLMs lack sufficient biomedical expertise, while English-focused training limits their effectiveness in Chinese medical contexts.
Method
HuaTuo fine-tunes the open-source LLaMA-7B model using structured and unstructured CMeKG knowledge and generated knowledge-based instruction data.
Results
HuaTuo significantly improves knowledge usability without much compromising safety in Chinese medical dialogue evaluation.
Takeaways & Limitations
The study presents an open-source Chinese biomedical LLM tuned with medical knowledge-based instruction data.
Takeaways & Limitations
The accuracy of responses generated by large language models cannot be guaranteed, and the medical knowledge is not a substitute for professional medical advice.
Abstract
from arXiv · showhide
Large Language Models (LLMs), such as the LLaMA model, have demonstrated their effectiveness in various general-domain natural language processing (NLP) tasks. Nevertheless, LLMs have not yet performed optimally in biomedical domain tasks due to the need for medical expertise in the responses. In response to this challenge, we propose HuaTuo, a LLaMA-based model that has been supervised-fine-tuned with generated QA (Question-Answer) instances. The experimental results demonstrate that HuaTuo generates responses that possess more reliable medical knowledge. Our proposed HuaTuo model is accessible at https://github.com/SCIR-HI/Huatuo-Llama-Med-Chinese.
1 Introduction
HuaTuo addresses the limitations of general-domain LLMs in biomedical and Chinese-language contexts by adapting LLaMA with knowledge-based instruction data. The paper also introduces SUS to evaluate biomedical responses across safety, usability, and smoothness.
- General-domain LLM knowledge can be insufficient for biomedical tasks requiring accurate, specialized expertise.The paper highlights risks involving diagnostic precision, drug recommendations, and medical advice.
- English-focused training constrains LLM comprehension and responses in Chinese contexts.
- HuaTuo adapts the open-source LLaMA-7B model for Chinese biomedical applications using structured and unstructured CMeKG knowledge.
- Over 8,000 knowledge-based instruction instances were collected for supervised fine-tuning.
- SUS evaluates biomedical language models along safety, usability, and smoothness.Safety concerns harmful misinformation, usability reflects medical expertise, and smoothness reflects language-model ability.
2 Related Works
Related work shows that LLMs provide strong general-domain capabilities but remain constrained by limited biomedical and Chinese-specific knowledge. Existing biomedical adaptations commonly use ChatGPT-assisted synthetic, distilled, or translated data to train smaller models.
- LLMs exhibit emergent abilities and strong general-domain performance, but their biomedical specialization remains limited.
- LLaMA is an open-source alternative to proprietary GPT-style models, while Alpaca applies instruction tuning to LLaMA.
- LLaMA performs subpar on Chinese tasks because its training data is primarily English, motivating Chinese-oriented models such as GLM and ChatGLM.
- Biomedical adaptations such as Chatdoctor and DoctorGLM use ChatGPT-synthesized or translated conversational data for fine-tuning.
- Prior biomedical adaptations illustrate that LLMs can be applied to specialized medical settings.
3 HuaTuo Model
HuaTuo is built by fine-tuning LLaMA-7B with Chinese medical knowledge and generated question-answer data. The data-generation process prioritizes factual correctness by sampling knowledge from CMeKG and producing more than 8,000 training instances.
- 3.1 Base Model: The model uses the open-source LLaMA-7B base for accessible training.
- 3.2 Medical Knowledge: CMeKG supplies Chinese medical knowledge concerning diseases, drugs, symptoms, and related entities.
- 3.3 Knowledge-based Instruction Data: Instruction data are generated from sampled knowledge-graph instances, emphasizing factual correctness in biomedical responses.
- 3.3 Knowledge-based Instruction Data: For medical dialogue, HuaTuo discards generic instructions and preserves the question inputs.
- 3.3 Knowledge-based Instruction Data: More than 8,000 generated instruction instances are collected as supervised-fine-tuning training data.
4 Experiment
The experiments compare HuaTuo with LLaMA, Alpaca, and ChatGLM using biomedical-oriented evaluation criteria. Results indicate that HuaTuo improves medical knowledge usability without substantially sacrificing safety.
- 4.1 Baselines: HuaTuo is evaluated against four baseline models, including LLaMA, Alpaca, and ChatGLM.
- 4.2 Metrics: Medical QA evaluation measures safety, usability, and smoothness rather than only similarity to reference text.
- 4.3 Results: Five medically trained annotators score randomly mixed Chinese dialogue responses on a three-point SUS scale.
- 4.3 Results: HuaTuo significantly improves knowledge usability without much compromising safety.LLaMA has the highest safety score but often produces uninformative responses that rephrase questions.
Ethics Statement
HuaTuo is presented as a research initiative rather than a source of medical advice, and its generated responses are not guaranteed to be accurate or a substitute for professional care.
- HuaTuo is intended primarily for research and not for providing medical advice.
- The study’s medical information is sourced from an open-access medical knowledge graph.
- Generated responses cannot be guaranteed accurate and should not replace guidance from qualified medical professionals.