Source-linked AI summary
Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialogue
Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jia, Hongying Zan
TL;DR
Chinese medical LLMs have been limited by dependence on SFT, single-turn or distilled dialogues, and insufficient proactive multi-turn inquiry. Zhongjing combines continuous pre-training, SFT, and RLHF with the 70,000-instance CMtMedQA dataset and medical-specific evaluation, outperforming open-source Chinese medical LLMs across dimensions and matching ChatGPT in some areas. The authors caution that it cannot guarantee accurate responses and should remain a research tool rather than a substitute for professional consultation.
Problem
Existing Chinese medical LLMs rely on SFT with single-turn or distilled dialogue data, limiting proactive inquiry, multi-turn comprehension, and alignment with expert intentions.
Method
Zhongjing uses continuous pre-training, SFT, and RLHF, supplemented by CMtMedQA, a 70,000-instance real multi-turn dialogue dataset covering 14 departments.
Results
Zhongjing surpasses other open-source Chinese medical LLMs across capacity dimensions and matches ChatGPT in some areas despite having 1% of its parameters.
Takeaways & Limitations
CMtMedQA improves complex multi-turn dialogue and proactive inquiry, while pre-training and RLHF contribute to medical knowledge, instruction following, and safety.
Takeaways & Limitations
Zhongjing cannot guarantee accurate responses, so users should exercise caution and seek expert advice; it remains mainly a research tool rather than a replacement for professional consultation.
Abstract
from arXiv · showhide
Recent advances in Large Language Models (LLMs) have achieved remarkable breakthroughs in understanding and responding to user intents. However, their performance lag behind general use cases in some expertise domains, such as Chinese medicine. Existing efforts to incorporate Chinese medicine into LLMs rely on Supervised Fine-Tuning (SFT) with single-turn and distilled dialogue data. These models lack the ability for doctor-like proactive inquiry and multi-turn comprehension and cannot align responses with experts' intentions. In this work, we introduce Zhongjing, the first Chinese medical LLaMA-based LLM that implements an entire training pipeline from continuous pre-training, SFT, to Reinforcement Learning from Human Feedback (RLHF). Additionally, we construct a Chinese multi-turn medical dialogue dataset of 70,000 authentic doctor-patient dialogues, CMtMedQA, which significantly enhances the model's capability for complex dialogue and proactive inquiry initiation. We also define a refined annotation rule and evaluation criteria given the unique characteristics of the biomedical domain. Extensive experimental results show that Zhongjing outperforms baselines in various capacities and matches the performance of ChatGPT in some abilities, despite the 100x parameters. Ablation studies also demonstrate the contributions of each component: pre-training enhances medical knowledge, and RLHF further improves instruction-following ability and safety. Our code, datasets, and models are available at https://github.com/SupritYoung/Zhongjing.
1 Introduction
Zhongjing addresses limitations in Chinese medical LLMs by combining a full training pipeline with authentic multi-turn doctor-patient dialogues and medical-specific evaluation. The resulting model surpasses other open-source Chinese medical LLMs across capacity dimensions and matches ChatGPT in some areas.
- Motivation: Biomedical applications require specialized knowledge, high accuracy, and safety, but general LLMs often perform worse in these domains.
- Motivation: Existing Chinese medical models rely mainly on SFT and single-turn or distilled dialogue data, limiting proactive doctor-like inquiry and multi-turn interaction.
- Approach: Zhongjing implements continuous pre-training, SFT, and RLHF, with SFT using single-turn, multi-turn, NLP-task, and general-dialogue data.
- Results: Zhongjing surpasses other open-source Chinese medical LLMs across capacity dimensions and matches ChatGPT in some areas despite having 1% of its parameters.
- Approach: CMtMedQA contains 70,000 real medical dialogue instances from 14 departments, including numerous proactive doctor inquiries.
- Evaluation: Zhongjing is evaluated across three capacity dimensions and nine abilities using an improved medical annotation and assessment framework.
2 Related Work
Related work improves biomedical LLMs through continued training, medical dialogue, and external knowledge, but Chinese medical modeling remains an active area of development.
- Biomedical LLMs: Biomedical LLMs such as MedAlpaca, ChatDoctor, Med-PaLM, and Med-PaLM2 use medical training strategies and receive favorable expert assessments in clinical response quality.
- Chinese medical LLMs: Chinese medical efforts include DoctorGLM, which combines extensive Chinese medical dialogues with an external medical knowledge base.
3 Methods
Zhongjing is constructed through continuous pre-training, SFT, and RLHF, using real medical corpora and multi-turn doctor-patient dialogues to develop medical knowledge, dialogue abilities, and safer responses.
- Overall Pipeline: Zhongjing’s method follows three stages: continuous pre-training, SFT, and RLHF.The method section presents these stages sequentially as the overall research workflow.
- Continuous Pre-training: Continuous pre-training combines diverse real medical sources, including textbooks, electronic health records, clinical records, and consultation dialogues.The corpus spans multiple departments and medical aspects to provide broad domain knowledge.
- Multi-turn Dialogue Dataset: CMtMedQA contains about 70,000 multi-turn dialogues and 400,000 conversations across 14 departments, with proactive inquiry statements.The dataset is sourced from real doctor-patient interactions and is designed to stimulate model-initiated medical inquiries.
- Supervised Fine-Tuning: SFT uses single-turn and multi-turn medical dialogues alongside medical NLP tasks and general dialogue data.The four data types target dialogue capability, generalization, and retention of prior general dialogue abilities.
- Reinforcement Learning from Human Feedback: RLHF ranks responses across 3 capability dimensions and 9 abilities, trains a reward model, and applies PPO to align outputs with expert intentions.Incorrect responses beyond the model’s capability are manually changed to boundary-aware replies such as “I’m sorry, I don’t know...”.
4 Experiments and Evaluation
The experiments compare Zhongjing with general and medical LLM baselines using resource-efficient training infrastructure and report converged training losses.
- Implementation: Training uses 4 A100-80G GPUs, LoRA during non-pretraining stages, fp16 precision, ZeRO-2, and gradient accumulation.These choices are described as supporting parameter-efficient tuning and balanced training costs.
- Training: Training losses for all stages converged within an effective range.
- Baselines: The experiments compare Zhongjing with LLM baselines spanning different parameter scales, including general and medical models.The baseline set includes ChatGPT, Ziya-LLaMA, and BenTsao.
Evaluation
Zhongjing is evaluated on separate multi-turn and single-turn Chinese medical dialogue test sets using three capability dimensions and nine abilities. It surpasses open-source baselines broadly, including HuatuoGPT across all three dimensions, while approaching ChatGPT in multi-turn dialogue.
- Evaluation Metrics: The assessment covers safety, professionalism, and fluency across nine specific abilities, using human experts and GPT-4.Human experts evaluate safety, while GPT-4 scores professionalism and fluency; results use Zhongjing’s win, tie, and loss rates.
- Results: Zhongjing achieves excellent performance on both single-turn and multi-turn dialogues across all three ability dimensions, surpassing baselines in most cases.The reported experiments are presented in Figures 4 and 5.
- Results: Zhongjing outperforms HuatuoGPT in all three ability dimensions despite using fewer fine-tuning instructions.The paper attributes this primarily to pre-training and RLHF, which provide foundational knowledge and boundary awareness.
- Results: Zhongjing shows superior multi-turn dialogue performance over all baselines except ChatGPT.The authors attribute this result to the CMtMedQA multi-turn dialogue dataset.
- Results: Instruction scale remains important, while distilled-data reliance is associated with poorer performance and targeted fine-tuning improves medical capabilities.BenTsao performs worst with 6k instructions; Zhongjing outperforms similarly sized DoctorGLM and exceeds the Ziya-LLaMA base model.
- Results: ChatGPT remains stronger in some cases, including safety, indicating that parameter size continues to matter for medical capabilities.The paper states Zhongjing can hold its ground against ChatGPT in most cases but falls behind in safety.
Ablation Study
Ablation experiments on CMtMedQA-test isolate continuous pre-training and RLHF, showing that both contribute to Zhongjing’s medical dialogue performance. RLHF produces the clearest gains in safety and response length.
- Experimental Design: Ablations compare models with and without pre-training or RLHF on safety, professionalism, fluency, and response length.The experiments use the evaluation strategy defined for the CMtMedQA test dataset.
- Findings: Pre-training and RLHF improve medical LLM performance, with RLHF producing the most significant gains in safety and response length.The authors interpret these gains as evidence that RLHF aligns the model more closely with medical experts.
- Findings: RLHF reduces dangerous and toxic responses while improving output quality and information content.Improvements in fluency and professionalism are relatively small because the preceding model already had high medical performance.
Case Study
The case study tests Zhongjing on a challenging multi-turn question requiring proactive inquiry and deep medical understanding. Its response identifies plausible causes and provides specific medication-related advice, demonstrating practical strengths over the baselines.
- Case Study Setup: The case study targets a challenging question combining multi-turn dialogue, proactive inquiry, and deep medical understanding.Responses from four baseline models are provided in the appendix for comparison.
- Case Study Findings: Zhongjing identifies potential causes such as allergic dermatitis or drug eruption and gives specific advice about stopping or changing medications.The example is presented as evidence of the model’s professional capabilities and practical value.
5 Conclusion and Limitations
The paper concludes that Zhongjing combines a full pre-training-to-RLHF pipeline with the CMtMedQA multi-turn dataset and outperforms other open-source Chinese medical LLMs. The authors caution that it remains a research tool, not a substitute for professional consultation.
- Conclusion: Zhongjing implements the full training pipeline from pre-training through SFT to RLHF and outperforms other open-source Chinese medical LLMs.Additional experiments highlight the significance of pre-training and RLHF for the medical domain.
- Conclusion: CMtMedQA is a large-scale Chinese multi-turn medical dialogue dataset introduced alongside Zhongjing.
- Limitations: Zhongjing cannot guarantee accuracy for every response, so users should treat generated medical information cautiously and seek expert advice.
- Limitations: The model remains mainly a research tool rather than a replacement for professional medical consultation.Future work targets safety, additional real-user data, and non-textual multimodal information.
A Evaluation Template
The evaluation template compares two responses to the same medical question using prioritized criteria for safety, professionalism, and fluency, with GPT-4 producing a Win, Lose, or Tie judgment.
- The template evaluates two responses to the same medical question as Answer1 relative to Answer2.
- Safety requires accurate medical knowledge, acknowledging unknowns, avoiding harmful suggestions, and respecting medical ethics and patient choices.
- Professionalism measures question understanding, relevant answers, clear explanations, and proactive inquiry into the patient’s condition when necessary.
- Fluency requires semantic coherence, logical consistency, stylistic consistency, and a friendly answering attitude.
- Criteria are prioritized as Safety > Professionalism > fluency, and the output is Win, Lose, or Tie.
B Training Settings in every stage
The paper provides parameter settings for each training phase.
- Table 4 lists parameter settings for each training phase.
C Dialogue Cases
The dialogue cases compare baseline outputs with Zhongjing’s multi-turn dialogue and inquiry behavior. Baselines receive single-turn evaluation because they lack proactive inquiry ability.
- Table 5 presents outputs from four baseline models for the same query and evaluates them only in a single-turn setting.
- Table 6 showcases Zhongjing’s ability to conduct multi-turn dialogue and initiate inquiry.