Source-linked AI summary
HuatuoGPT, towards Taming Language Model to Be a Doctor
Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Jianquan Li, Guiming Chen, Xiangbo Wu, Zhiyi Zhang, Qingying Xiao, Xiang Wan, Benyou Wang, Haizhou Li
TL;DR
Medical LLMs need stronger medical, doctor-like, and localized capabilities than general ChatGPT provides. HuatuoGPT combines ChatGPT-distilled data with doctors’ real-world data in SFT, uses RLAIF to align both strengths, and evaluates performance automatically and manually. It achieves strong results across GPT-4 evaluation, human evaluation, and medical benchmarks, while remaining not ready for unrestricted real-world biomedical deployment because generated content is difficult to verify.
Problem
ChatGPT and GPT-4 show relatively poor performance in medicine, while general models lack extensive customization for localized medical practices such as Chinese medicine.
Method
HuatuoGPT combines ChatGPT-distilled and doctors’ real-world instruction and conversation data, then applies RLAIF to align patient-friendly and doctor-like responses.
Results
HuatuoGPT outperforms incumbent Chinese medical models in GPT-4 evaluation, surpasses GPT-3.5-turbo in most evaluated cases, and achieves SOTA performance on several medical benchmarks.
Takeaways & Limitations
Combining complementary real-world and distilled data with AI feedback produces a conversational healthcare model that integrates presentation quality with medical reliability.
Takeaways & Limitations
Generated medical content remains difficult to verify for accuracy, so these models are not yet fully prepared for real-world biomedical deployment.
Abstract
from arXiv · showhide
In this paper, we present HuatuoGPT, a large language model (LLM) for medical consultation. The core recipe of HuatuoGPT is to leverage both \textit{distilled data from ChatGPT} and \textit{real-world data from doctors} in the supervised fine-tuned stage. The responses of ChatGPT are usually detailed, well-presented and informative while it cannot perform like a doctor in many aspects, e.g. for integrative diagnosis. We argue that real-world data from doctors would be complementary to distilled data in the sense the former could tame a distilled language model to perform like doctors. To better leverage the strengths of both data, we train a reward model to align the language model with the merits that both data bring, following an RLAIF (reinforced learning from AI feedback) fashion. To evaluate and benchmark the models, we propose a comprehensive evaluation scheme (including automatic and manual metrics). Experimental results demonstrate that HuatuoGPT achieves state-of-the-art results in performing medical consultation among open-source LLMs in GPT-4 evaluation, human evaluation, and medical benchmark datasets. It is worth noting that by using additional real-world data and RLAIF, the distilled language model (i.e., HuatuoGPT) outperforms its teacher model ChatGPT in most cases. Our code, data, and models are publicly available at \url{https://github.com/FreedomIntelligence/HuatuoGPT}. The online demo is available at \url{https://www.HuatuoGPT.cn/}.
1 Introduction
Medicine is an experience-driven, language-centered domain where LLMs could expand access, but ChatGPT has important medical and localization limitations. HuatuoGPT combines distilled ChatGPT data with doctors’ real-world data, RLAIF, and systematic evaluation to improve medical consultation.
- Problem: ChatGPT performs relatively poorly in medicine, especially Chinese, and does not reliably diagnose, prescribe, ask clarifying questions, or provide doctor-like specialized responses.The paper also identifies hallucination as a challenge associated with autoregressive generation.
- Motivation: LLMs could support medical advice, triage, diagnosis, prescribing, report interpretation, and online consultation, while potentially reducing hospital crowding and psychological barriers to seeking care.Online platforms may provide scalable, more comfortable access to medical discussion and help alleviate pressure on hospitals.
- Motivation: Healthcare inequality in China is substantial, with average life expectancy reported as approximately 82 years in Shanghai versus 73 years in Guizhou.The paper presents this disparity as motivation for developing localized medical LLMs.
- Approach: HuatuoGPT trains with both ChatGPT-distilled and doctors’ real-world instruction and conversation data, then uses RLAIF to reward patient-friendly and doctor-like responses.Distilled data supports fluent, well-presented instruction following, while doctor data contributes medical knowledge and consultation behaviors.
- Results: HuatuoGPT outperformed incumbent Chinese medical models in GPT-4 evaluation, surpassed GPT-3.5-turbo in most evaluated cases, exceeded ChatGPT in over 60% of complex multi-turn cases, and achieved SOTA on several benchmarks.The evaluation combined automated and manual assessments; manual results corroborated the automated findings.
2 Motivations
The paper argues that ChatGPT and doctors provide complementary strengths for medical consultation: ChatGPT offers fluent, informative responses, while doctors provide contextual questioning and accurate diagnosis. Training solely on either source leaves important weaknesses, motivating a hybrid approach.
- Learning from Humans or ChatGPT?: ChatGPT-distilled data provides fluent dialogue and instruction following but may contain incorrect or ambiguous medical information.Distillation can quickly provide diverse medical dialogue and seamless conversational skills, but its quality can fluctuate.
- Learning From Both Doctors and ChatGPT in Medicine: Doctors’ data and ChatGPT data are therefore complementary: the former contributes medical expertise and diagnosis, while the latter contributes detailed, coherent presentation.The paper summarizes this complementarity as a difference in behavior across multiple aspects of medical consultation.
- Learning from Humans or ChatGPT?: Purely human conversation data can be accurate and professional but often produces short, poorly presented, and uninformative responses.Doctor responses are personalized and diagnostic, yet time constraints can make them informal, concise, or incoherent.
- Learning from Humans or ChatGPT?: ChatGPT often gives general advice without asking clarifying questions or conducting the interactive diagnosis that doctors perform.The paper illustrates this contrast with abdominal-pain dialogue, where the doctor asks about location, severity, and duration.
- Learning From Both Doctors and ChatGPT in Medicine: HuatuoGPT combines distilled and real-world data to produce responses that are both patient-friendly and doctor-like, including clarifying questions and accurate diagnosis.The proposed training strategy mixes both sources during supervised fine-tuning and further uses AI feedback.
3 Methodology
HuatuoGPT uses a two-stage strategy that first fine-tunes on hybrid ChatGPT-distilled and real-world doctor data, then applies reinforcement learning from AI feedback. The data pipeline combines instruction and conversation examples, while the reward model favors informative, coherent, accurate, and doctor-like responses.
- Overall Training Strategy: HuatuoGPT first applies supervised fine-tuning to hybrid data, then reinforces desired responses with AI feedback.This two-stage design aims to combine doctors’ inquiry and diagnostic behavior with language models’ rich and coherent responses.
- SFT with Hybrid Data: The hybrid SFT corpus contains both instruction and conversation data from ChatGPT and real-world doctors.The two data types are intended to provide instruction following, fluent dialogue, interactive diagnosis, medical knowledge, and doctor-like behavior.
- SFT with Hybrid Data: ChatGPT-distilled instructions are constructed with a taxonomy and self-instruct process to generate diverse medical instructions across roles and use cases.Seed instructions are organized by role or use case before being mixed for further self-instruct generation.
- SFT with Hybrid Data: Real-world doctor instructions are refined from authentic doctor-patient question-answer pairs to improve their quality and reliability.The source responses are described as expert, relevant, and concise.
- SFT with Hybrid Data: Distilled conversations are generated by two role-conditioned ChatGPT instances using patient backgrounds and doctor diagnoses informed by a medical diagnosis database.Real-world conversations are collected from genuine scenarios and refined with language models because they can be overly concise or colloquial.
- RL with AI Feedback: The reward model scores sampled responses for qualities including informativeness, coherence, human preference adherence, and factual accuracy against doctors’ diagnoses.During reinforcement learning, the final reward combines the reward-model score with a KL penalty that limits deviation from the initial policy; λKL is set to 0.05.
4 Experiments
The experiments evaluate HuatuoGPT on Chinese medical benchmarks, GPT-4-reviewed single- and multi-turn consultations, and manual medical-consultation criteria. Across these settings, it generally outperforms the compared open-source and fine-tuned baselines, with strong diagnosis and interactive performance.
- Evaluation setup: HuatuoGPT is evaluated with automatic and manual assessments spanning Chinese QA benchmarks, GPT-4 reviews, and medical-consultation criteria.The automated evaluations cover single-turn questions and multi-turn conversations, while manual evaluation considers diagnosis accuracy, treatment recommendations, and medication knowledge.
- Medical benchmarks: The benchmark comparison uses cMedQA2, webMedQA, and Huatuo-26M, against zero-shot ChatGPT and fine-tuned Chinese T5.cMedQA2 contains 108,000 questions and 203,569 answers; webMedQA contains 63,284 questions; Huatuo-26M contains 26M QA pairs.
- Medical benchmarks: HuatuoGPT outperforms fine-tuned T5 on cMedQA2 and webMedQA while maintaining high accuracy, fluency, and diversity across metrics.The result suggests robust generalization across Chinese medical question-answering tasks.
- GPT-4 evaluation: HuatuoGPT surpasses BenTsao, DoctorGLM, and GPT-3.5-turbo overall on GPT-4-reviewed single-turn questions, but remains below GPT-4 in almost all categories.It exceeds GPT-3.5-turbo in Indicators Interpretation, Condition Diagnosis, and Medical Expenses, and performs similarly in Efficacy and Disease Description.
- Manual evaluation: HuatuoGPT wins 86% of manually evaluated multi-turn cases against DoctorGLM and 58% against ChatGPT.The authors associate this result with stronger interactive diagnostic capability in patient consultations.
5 Discussion
The ablations show that distilled and real-world doctor data contribute different strengths, while RLAIF helps combine patient-friendly presentation with doctor-like interaction. The authors nevertheless caution that generated medical consultation is not ready for unrestricted real-world deployment.
- Ablation Study: Real-world data makes HuatuoGPT more likely to ask clarifying questions, whereas distilled data produces detailed, well-organized, informative responses.The real-data model is briefer and less organized, while the distilled-data model focuses more on suggestions than diagnostic decisions.
- Ablation Study: RLAIF enables follow-up questioning for more accurate diagnoses, unlike the variant trained without RLAIF.The authors suggest the no-RLAIF model may be biased toward ChatGPT data while overlooking real-world data.
- Limitation: Generation-based medical consultation remains constrained by the difficulty of verifying generated content’s accuracy and correctness.The authors highlight severe ethical implications when misleading medical information is disseminated.
- Limitation: The authors recommend caution before real-world deployment and call for improved accuracy checking and error correction.They state that further research and development are necessary to refine these models and reduce associated risks.
6 LLMs in Medicine
The paper situates HuatuoGPT within the development of biomedical and medical consultation language models, including domain-pretrained, instruction-tuned, retrieval-enabled, multimodal, and Chinese systems. Existing models differ in backbone, training data, and capabilities.
- Domain-pretrained models: BioMedLM is a biomedical language model continued from GPT-2 and trained on PubMed Abstracts and PubMed Central data.Its training corpus contains around 50B tokens, spanning approximately 16 million abstracts and 5 million full-text articles.
- Instruction-tuned models: Instruction-tuned medical models include MedAlpaca, ChatDoctor, Baize-healthcare, and Med-PaLM2, using curated, internet, real-world, retrieved, or multimodal medical data.Their approaches include reformatted medical NLP tasks, online consultation conversations, model-generated biomedical examples, retrieval, and visual-modality integration.
- Comparison: The paper compares popular medical language models by their training data sources and model specifications.The comparison is summarized in Table 10.
- Chinese medical models: Chinese medical systems include DoctorGLM, which combines translated ChatDoctor data with Chinese medical dialogues, and BenTsao, trained on over 8K ChatGPT-generated instructions from CMeKG.These systems are built on ChatGLM and LLaMA-based architectures, respectively.
7 Conclusion
The paper concludes that combining complementary real-world and distilled data with AI-feedback reinforcement learning supports a reliable, conversational healthcare model. It presents this approach as a basis for further progress in medical AI and patient outcomes.
- Conclusion: HuatuoGPT combines real-world and distilled data with AI-model feedback through reinforcement learning.The approach is intended to address limitations of relying solely on real or synthetic data while combining their strengths.
- Conclusion: The authors conclude that continual feedback-based refinement can improve conversational abilities while maintaining reliability for healthcare applications.They identify further research as having potential to advance AI in healthcare and improve patient outcomes.
A.1 Distilled Instructions from ChatGPT
The distilled-instruction process uses ChatGPT to create diverse, medically grounded role–instruction–input triplets and role-enhanced prompts for generating doctor-like responses.
- Distilled instruction generation: Self-instruction generates 20 diverse role, instruction, and input triplets from manually constructed medical seed instructions.Roles may be specialized by medical department, while instructions and inputs should be varied and substantive.
- Triplet structure: The triplets specify a medical role, an instruction for improving work efficiency, and corresponding substantive input.Instructions are generally one or two sentences and may contain placeholders filled by the input field.
- Role enhancement: Role-enhanced instructions ask an experienced medical professional to respond patiently, warmly, comprehensively, and with clarifying questions when information is insufficient.The prompt also requests additional advice at the end of the diagnosis.
A.2 Real-world Instructions from Doctors
The real-world instruction pipeline collects high-quality medical question–answer pairs from the web, refines them with LLMs, and prompts HuatuoGPT to produce detailed doctor-like replies.
- Data collection: Real-world question-answering data are collected from the web and sampled as high-quality pairs for training.Each pair is refined by LLMs using a specified prompt.
- Prompt inputs: The refinement prompt supplies a patient question and a doctor-response reference to guide answer generation.The patient question and reference response are represented as separate prompt fields.
- Response refinement: HuatuoGPT is instructed to answer in a gentle, doctor-like, detailed, and helpful manner, asking for more information when diagnosis is unclear.The prompt uses the reference response while allowing additional questioning when necessary.
A.3 Real-world Conversations with Doctors
The real-world conversation setup uses separate patient and doctor prompts to simulate multi-turn consultations, while an evaluator rates AI responses against standard answers.
- Patient simulation: The patient prompt frames consultation as a multi-turn interaction and asks the patient to keep each question concise.The simulated patient ends the consultation by saying “再见” when it considers the interaction complete.
- Doctor simulation: The doctor prompt requests patient, comprehensive, gentle, and detailed responses, with follow-up questions when information is insufficient for diagnosis.It also instructs the doctor model to provide additional detailed advice after diagnosis.
- Doctor simulation: The doctor prompt restricts the model to patients’ textual descriptions and excludes images or other attachments.When diagnosis remains unclear, the model should ask for more medical information.
- Response evaluation: An evaluation prompt rates two AI responses from 1 to 10 on accuracy, conciseness, and similarity to a standard answer using conversation context.The ratings are requested in a specified output format.