PaperlayerView exact revision on arXiv

Source-linked AI summary

DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation

Zhijie Bao, Wei Chen, Shengze Xiao, Kuang Ren, Jiaao Wu, Cheng Zhong, Jiajie Peng, Xuanjing Huang, Zhongyu Wei

arXiv:2308.14346v1cs.CLcs.AI

TL;DR

Prior healthcare systems often focus on specific tasks or diseases, limiting applicability to end-to-end consultation. DISC-MedLLM builds SFT datasets using medical knowledge graphs, reconstructed dialogues, and human-guided preference rephrasing, and reports gains over medical LLMs in single- and multi-turn consultation.

  • Problem

    Prior healthcare studies focus mainly on specific tasks or diseases, limiting applicability and leaving a gap in comprehensive, end-to-end conversational medical services.

    Source: Introduction · p. 1

  • Method

    DISC-MedLLM uses SFT datasets built with medical knowledge graphs, real-world dialogue reconstruction, and human-guided preference rephrasing.

    Source: Conclusion · p. 9

  • Results

    Over 10% higher on average than HuatuoGPT (13B) in medical multiple-choice questions, DISC-MedLLM also shows superior overall performance to GPT-3.5, HuatuoGPT, and BianQue in simulated consultations.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The evaluations support DISC-MedLLM's effectiveness in multiple-choice Q&A and systematic medical consultations.

    Source: Conclusion · p. 9

    This version uses Baichuan as its base model, although the authors state the strategy can apply to all decoder-only foundation models.

    Source: Introduction · p. 2

Abstract

from arXiv · show

We propose DISC-MedLLM, a comprehensive solution that leverages Large Language Models (LLMs) to provide accurate and truthful medical response in end-to-end conversational healthcare services. To construct high-quality Supervised Fine-Tuning (SFT) datasets, we employ three strategies: utilizing medical knowledge-graphs, reconstructing real-world dialogues, and incorporating human-guided preference rephrasing. These datasets are instrumental in training DISC-MedLLM, surpassing existing medical LLMs in both single-turn and multi-turn consultation scenarios. Extensive experimental results demonstrate the effectiveness of the proposed model in bridging the gap between general language models and real-world medical consultation. Additionally, we release the constructed dataset and model weights to further contribute to research and development. Further details and resources can be found at https://github.com/FudanDISC/DISC-MedLLM

Introduction

DISC-MedLLM targets the gap between task-specific medical systems and end-to-end consultation by training a general Chinese LLM on SFT data designed to encode reliable medical knowledge and real-world dialogue behavior. The paper reports gains over medical LLM baselines in single-turn and simulated multi-turn evaluations.

Introduction

  • Prior intelligent healthcare systems often target specific tasks or diseases, while medical consultations require reliable knowledge and multi-turn inquiry that general LLMs may lack.The paper identifies hallucination and irrelevant case-specific responses, as well as limited ability to inquire across turns before advising, as challenges for general-domain LLMs.Source: Introduction · p. 1
  • The SFT data combines knowledge-graph-driven QA, GPT-3.5 reconstruction of real-world consultations, and human-guided preference rephrasing, yielding 50k, 420k, and 2k samples, respectively.The knowledge triples are sampled by department using real consultation query distributions; reconstructed forum dialogues address informal language, inconsistent terminology, and variation in professional expression.Source: Introduction · pp. 1–2
  • The datasets train DISC-MedLLM in two stages on a 13B Chinese base model, with single-turn multiple-choice accuracy and simulated multi-turn consultation evaluated separately.This version uses Baichuan, though the strategy is described as applicable to decoder-only foundation models; multi-turn tests use GPT-3.5 as patient and GPT-4 to assess proactivity, accuracy, helpfulness, and linguistic quality.Source: Introduction · p. 2
  • DISC-MedLLM exceeds 13B HuatuoGPT (Zhang et al. 2023) by over 10% on average on medical multiple-choice questions, trails GPT-3.5, and leads Chinese medical LLMs in specified consultation scenarios.In simulated consultations, it also outperforms GPT-3.5, HuatuoGPT, and BianQue (Chen et al. 2023d) overall; it achieves the best performance among Chinese medical LLMs in scenarios involving medical departments and patient intents.Source: Introduction · p. 2

Advantages of DISC-MedLLM

DISC-MedLLM’s fine-tuning dataset draws on medical knowledge graphs and real doctor-patient consultations, using LLM and human involvement to support reliable knowledge, multi-turn inquiry, and alignment with human preferences.

Advantages of DISC-MedLLM

  • DISC-Med-SFT combines knowledge-graph QA, reconstructed doctor-patient dialogues, and behavioral-preference data, with department-based sampling and LLM and human involvement.LLMs paraphrase real-world conversations, while humans guide alignment with high-quality conversational behaviors.Source: Figure 2 · p. 3, Advantages of DISC-MedLLM · p. 3

Knowledge Intensive and Reliable

  • Medical knowledge comes from human-labeled knowledge graphs and doctor-authored dialogues, while ChatGPT rewrites rather than generates it.Paraphrases retain the underlying medical knowledge while ChatGPT can elaborate on non-essential information, such as causes of a condition.Source: Knowledge Intensive and Reliable · p. 3

Ability of Multi-turn Inquiry

  • Real-world multi-turn doctor-patient conversations train DISC-MedLLM to ask meaningful questions and clarify symptoms that patients may struggle to describe all at once.The paper contrasts this with single-turn, template-like responses that can overlook inquiry and clarification.Source: Ability of Multi-turn Inquiry · p. 3

Alignment with Human Preference

  • DISC-MedLLM combines human-preference guidance with ChatGPT behavioral patterns to produce comprehensive, empathetic responses aligned with real-world consultation distributions.The paper frames doctors as mentors for medical knowledge and decision-making, and ChatGPT as a mentor for behavioral patterns and human preferences, with precise sampling and human intervention.Source: Alignment with Human Preference · pp. 3–4

DISC-Med-SFT

DISC-Med-SFT combines medical Q&A and consultation data with general-domain samples. Its construction draws on real-world dialogue reconstruction, knowledge-graph QA generation, and human-guided behavior tuning.

DISC-Med-SFT

  • DISC-Med-SFT contains over 470k medical examples across single-turn Q&A, multi-turn consultations, and multiple-choice Q&A, plus over 34k general-domain samples.The dataset is derived from existing medical datasets and includes general conversation and instruction data.Source: DISC-Med-SFT · p. 4

Real-world Dialogue Records

  • MedDialog (Chen et al. 2020b) and cMedQA2 supplied 400k and 20k filtered source samples, respectively, for real-world dialogue construction.MedDialog contains over 3 million multi-turn consultations; cMedQA2 contains 108k single-turn consultation and advisory conversations.Source: Real-world Dialogue Records · p. 4
  • GPT-3.5 reconstructed dialogues to standardize language, preserve and expand doctors’ key information, and remove responses unsuitable for an AI doctor.The resulting responses align with the AI medical assistant identity while providing more comprehensive assistance; examples are shown in Figure A2.Source: Real-world Dialogue Records · p. 4

Knowledge Graph QA pairs

  • Over 50k medical-scenario conversations were generated from a Chinese medical knowledge graph covering over 10k diseases, nearly 20k medications, and more than 10k symptoms.GPT-3.5 first converted sampled knowledge into natural-language QA pairs, then turned them into diverse single-turn medical dialogues; disease sampling followed MedDialog’s department distribution.Source: Knowledge Graph QA pairs · p. 5

Human Preferences Guided Conversation Samples

  • About 2,000 unused MedDialog and cMedQA2 samples were selected and reconstructed to produce human-supervised behavior-tuning examples.GPT-4 reconstructed selected consultation examples, which were manually revised and provided as few-shot demonstrations to GPT-3.5 for generating 2,000 samples.Source: Human Preferences Guided Conversation… · p. 5

Others

  • MedMCQA yielded about 8k Chinese medical QA samples, while moss-sft-003 and alpaca gpt4 data zh contributed 33k and 1k general-domain samples.MedMCQA questions and answers were refined with explanations and translated into Chinese, with approximately 2k samples retaining multiple-choice format; general data were included to mitigate foundational capability degradation during medical-focused SFT.Source: Others · p. 5

Training Details

DISC-MedLLM is built on Baichuan-13B-Base and trained through two SFT stages: one for medical knowledge and dialogue capability, and one for alignment with human preferences.

Training Details

  • The model is developed on Baichuan-13B-Base, an open-source LLM with over 13.2 billion parameters trained on 1.4 trillion tokens.The base model is described as performing well in both English and Chinese.Source: Training Details · p. 5
  • Training uses two SFT stages, both completed on 4 A800 GPUs.Stage 1 builds domain knowledge and medical dialogue capabilities; Stage 2 aligns behavioral patterns and usefulness with human preferences.Source: Training Details · p. 6
  • Stage 1 uses large-scale data to impart medical domain knowledge and dialogue capability.Training combines a 420k adapted AI doctor-patient dialogue dataset, 50k knowledge-graph QA pairs, MedMCQA, and moss-sft-003; settings include global batch size 24, learning rate 1e-5, AdamW, 1 epoch, maximum sequence length 2048 tokens, 1800 warm-up steps, and no weight decay.Source: Training Details · p. 6
  • Stage 2 trains the model to align with human preferences for behavioral patterns and usefulness in medical dialogue.It combines a 2k preference-aligned behavioral dataset with 1k Alpaca GPT4 data zh data; settings include global batch size 8, learning rate 5e-6, AdamW, 1 epoch, maximum sequence length 2048 tokens, and no weight decay.Source: Training Details · p. 6

Evaluation Setup

The evaluation covers single-turn multiple-choice medical question answering and multi-turn medical conversations, using accuracy for the former and conversation-focused assessment for the latter.

Evaluation Setup

  • The evaluation tests medical LLMs in both single-turn question answering and multi-turn conversation.Source: Evaluation Setup · p. 6

Single-turn Question Answering Evaluation

Multiple-choice Datasets

  • The multiple-choice benchmark includes 1,362 MLEC-QA test questions and 270 NEEP questions from 2020 and 2021.MLEC-QA questions are randomly sampled from 10% of its test set; NEEP covers Western Medicine 306.Source: Multiple-choice Datasets · p. 6, Table 2 · p. 6
  • For few-shot evaluation, MLEC-QA examples come from its validation set and NEEP examples from its 2019 questions.Source: Multiple-choice Datasets · p. 6

GPT4-as-a-Judge

Dialogue Evaluation Datasets

  • Multi-turn evaluation uses 313 cases from CMB-Clin, CMD, and CMID, spanning real-world consultations, medical specialties, and user intentions.The set comprises 73 CMB-Clin cases, 120 CMD questions, and 120 CMID queries.Source: Dialogue Evaluation Datasets · p. 6

Evaluation Metrics for Conversation

  • Conversation evaluation rates proactivity, factual accuracy, practical helpfulness, and linguistic quality.The criteria assess information gathering when details are insufficient, accurate advice, useful assistance, and natural expression.Source: Evaluation Metrics for Conversation · pp. 6–7

Evaluation Results

DISC-MedLLM is evaluated against general-purpose and Chinese medical LLMs in single-turn QA and multi-turn consultation. Its performance varies across evaluation datasets and tasks.

Comparative Models

  • The comparison includes GPT-3.5, GPT-4, Baichuan-13B-Chat, BianQue-2, and HuatuoGPT-13B; GPT-4 and BianQue-2 lack extensive multiple-choice testing for billing and output constraints.Source: Comparative Models · p. 7

Results for Single-Turn QA

  • GPT-3.5 leads multiple-choice assessment; DISC-MedLLM ranks second in few-shot and third behind Baichuan-13B-Chat in zero-shot, outperforming HuatuoGPT(13B).Source: Results for Single-Turn QA · p. 7

Results for Multi-turn Conversation

Discussion

  • Medical accuracy remains an unresolved concern because incorrect information has serious implications; retrieval augmentation faces barriers in repository curation and retriever-query alignment.Source: Discussion · p. 9

Related Works

Medical dialogue systems span many tasks, but limited applicability and integration hinder deployment. DISC-MedLLM instead emphasizes ChatGPT’s behavioral patterns and human preferences, with carefully constructed labels intended to mitigate hallucinations.

  • Medical dialogue systems address recognition, symptom identification, standardization, intent, report generation, state tracking, diagnosis, and drug recommendation, but limited scope and integration hinder deployment.Source: Related Works · p. 9
  • LLM distillation and self-instruction support end-to-end instruction-following systems and fine-tuning moderately sized pretrained models (Gou et al. 2021; Chen et al. 2022; Wang et al. 2022).Source: Related Works · p. 9
  • Medical LLMs include ChatDoctor, Baize-healthcare (Xu et al. 2023), MedAlpaca (Han et al. 2023), PMC-LLaMA (Wu et al. 2023), and Med-PaLM 2 (Singhal et al. 2023).ChatDoctor uses patient-doctor dialogues; Baize-healthcare trains on 100k ChatGPT-generated dialogues, while PMC-LLaMA fine-tunes on medical papers and Med-PaLM 2 shows potential for clinical use.Source: Related Works · p. 9
  • Chinese medical models include BenTsao (Wang et al. 2023a), DoctorGLM (Xiong et al. 2023), MedicalGPT (Xu 2023), ChatMed, and HuatuoGPT (Zhang et al. 2023).Their approaches include knowledge-based instruction data, incremental pre-training plus fine-tuning and reinforcement learning, ChatGPT-distilled consultation answers, and blends of real conversations with distilled data; ChatMed performs well in single-turn QA.Source: Related Works · p. 9
  • DISC-MedLLM focuses on learning ChatGPT’s behavioral patterns and human preferences rather than its medical knowledge, using carefully constructed labels to mitigate potential hallucinations.Source: Related Works · p. 9

Conclusion

DISC-MedLLM bridges general LLMs and real-world medical consultation through SFT datasets built with medical knowledge graphs, reconstructed dialogues, and human-guided preference rephrasing. The paper reports stronger performance than existing medical LLMs in single- and multi-turn consultations and releases its dataset and model weights.

Conclusion

  • DISC-MedLLM surpasses existing medical LLMs in single- and multi-turn consultations, with evaluations covering multiple-choice Q&A and systematic medical consultations.The paper presents the results as evidence of the model’s effectiveness.Source: Conclusion · p. 9
  • The SFT datasets use medical knowledge graphs, real-world dialogue reconstruction, and human-guided preference rephrasing to support accurate, truthful medical responses.The approach also leverages ChatGPT to rephrase existing medical NLP datasets for end-to-end conversational healthcare services.Source: Conclusion · p. 9
  • The authors release the constructed dataset and model weights to support further research and development.Source: Conclusion · p. 9
  • Future work will introduce retrieval-enhanced DISC-MedLLM to incorporate additional medical expertise for complex and rare medical cases.Source: Conclusion · p. 9

Appendix · Appendix B: Prompts using in our practices

Appendix B identifies prompts used to reconstruct real-world conversations, generate knowledge-graph QA pairs in two steps, and evaluate multi-turn conversations with GPT-4 as a judge.

Appendix B: Prompts using in our practices

  • Figure B1 presents the prompt for reconstructing real-world conversations.Source: Figure B1 · p. 16
  • Figures B2 and B3 present prompts for generating QA pairs from a knowledge graph in Step 1 and Step 2, respectively.Source: Figure B2 · p. 17, Figure B3 · p. 17
  • Figure B4 presents the prompt used for GPT-4-as-a-judge evaluation of multi-turn conversations.Source: Figure B4 · p. 18
Loading 2308.14346v1…